•  # 


*  •* 


■T  ••  « 


%  • 


v  »  „ 


Sc.i-'-OTE^t 

In  U 


Quality  Improvement 

Does  the  Air  Force 
Systems  Command 
Practice  What  It  Preaches 


John  A  Campbell,  Major,  USAF 


Approved  far  public  rofoaf} 
Distribution  Unlimited  •,  , 


REPORT  DOCUMEMTAT  }N  PAGE 


Form  Approved 
OMB  No.  0704-0188 


Public  reoemng  ou'oen  for  tni$  collection  of  information  estimates  to  average  1  hour  oer  resporse.  including  the  time  for  reviewing  instructions,  searching  e*  st'ng  data  sources, 
gathering  and  maintaining  the  data  reeded,  and  completing  and  reviewing  the  collection  of  information  Send  comments  regarding  tr  -s  burden  estimate  or  any  other  aspect  of  t"i$ 
collection  of  information,  .n;'uding  suggestions  for  reducing  this  ouraen  to  Washington  Headouarters  Services.  Directorate  for  information  Operations  and  Reports,  1215  JeHerson 
Davis  Highway.  Suite  '2C4.  Arlington,  V A  22202-4302.  and  to  the  Office  of  Management  and  Budget.  Paperwork  Reduction  Pruett  (0 70**-0 1 8t3).  Wash  ngton.  OC  20503 


1.  AGENCY  USE  ONLY  (Leave  biank )  2.  REPORT  DATE 


4.  TITLE  AND  SU3TITLE 


3. REPORT  TYPE  AND  DATES  COVERED 


Quality  Improvement:  Does  the  Air  Force  Systems  Command 
Practice  What  It  Preaches 


6.  AUTHOR(S) 

Major  John  A  Campbell,  USAF 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

AUCADRE/PTP 

Maxwell  AFB  AL  36112-5532 


9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 


8.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 


AU-ARI-88- 13 


10.  SPONSORING  /  MONITORING 
AGENCY  REPORT  NUMBER 


12a.  DISTRIBUTION /AVAILABILITY  STATEMENT  12b.  DISTRIBUTION  CODE 

Public  Release  a 


13.  ABSTRACT  (Maximum  200  words) 

The  objective  of  this  study  is  to  determine  if  Air  Force  Systems  Command  (AFSC) 
acquisition  personnel  are  aware  of  and  using  those  techniques  that  more  effectively 
identify  nonconformances  and  are  applying  techniques  to  reduce  incidences  of 
nonconformance.  Major  Campbell  uses  a  questionnaire.  He  surveys  program  directors, 
program  managers,  and  quality  assurance  personnel  at  the  AFSC's  five  major 
product  divisions. 


17.  SECURITY  CLASSIFICATION 
OF  REPORT 

UNCLAS 


\SN  75-10-01 -280-5503 


18.  SECURITY  CLASSIFICATION 
OF  THIS  PAGE 

UNCLAS 


19.  SECURITY  CLASSIFICATION 
OF  ABSTRACT 


15.  NUMBER  OF  PAGES 

192 


1b.  PRICE  CODE 
None 


20.  LIMITATION  OF  ABSTRACT 


UNCLAS 


Staroj’d  rorm  298  \8ev  2-89) 

P'**v.*  tyvj  t>y  2*9- '8 


Cut  along  dotted  line 


After  you  have  read  the  research  report,  please  give  us 
your  frank  opinion  on  the  contents.  All  comments — large 
or  small  complimentary  or  caustic — will  be  gratefully 
appreciated.  Mail  them  to:  CADRE/RI,  Building  1400, 
Maxwell  AFB  AL  361 12-5532. 


Quality  Improvement 

Does  the  Air  Force  Systems  Command 
Practice  What  It  Preaches? 


Campbell 


Thank  you  for  your  assistance 


Research  Report  No.  AU-AR1-88-13 


Quality  Improvement 

Does  the  Air  Force  Systems  Command 
Practice  What  It  Preaches ? 

by 


JOHN  A.  CAMPBELL,  Major,  USAF 

Research  Fellow 
Airpower  Research  Institute 


Air  University  Press 

Maxwell  Air  Force  Base,  Alabama  36112-5532 


March  1990 


This  study  represents  the  views  of  the  author  and  does  not 
necessarily  reflect  the  official  opinion  of  the  Air  University 
Center  for  Aerospace  Doctrine,  Research,  and  Education  (AUCADRE) 
or  the  Department  of  the  Air  force.  This  publication  has  been 
reviewed  by  security  anJ  policy  review  authorities  and  is  cleared 
for  public  release. 

This  document  is  th?  property  of  the  United  States  government 
and  is  not  to  be  reproduced  in  whole  or  in  part  without  perr.ission 
of  the  coirjnanaer,  AD  CADRE ,  Maxwell  Air  Force  base,  Alabama. 


i  i 


it 


Contents 


Chapter  Page 

DISCLAIMER  .  ii 

FOREWORD . xi 

ABOUT  THE  AUTHOR . xiii 

PREFACE  . xu 


1  INTRODUCTION .  1 

Quality  Regulations .  4 

Federal  Acquisition  Regulation . 4 

DOD  Supplement  to  the  Federal  Acquisition 

Regulation  .  4 

DODD  4155.1,  Quality  Program . 5 

AFR  74-1,  Quality  Assurance  Program . 5 

Mission  of  Air  Force  Systems  Command .  7 

Air  Force  Contract  Management  Division  . 7 

Armament  Division  .  8 

Aeronautical  Systems  Division  . 8 

Ballistic  Missile  Office . 8 

Electronic  Systems  Division . 8 

Space  Division  . 9 

Research  Objective  .  9 

Notes  .  9 


2 


QUALITY  ASSURANCE:  A  SURVEY  OF  PREVIOUS 

STUDIES . 

Quality  ’77 . 

Quality  Horizons . 

Contract  Incentives  for  Product  Quality  .... 

Project  Quality . 

Preproposal  Actions .  . 

Specification  and  Statement  of  Work 

Preparation . 

Negotiation  . 

Performance,  Administration  of  Contract  .  . 

Policy . 

Other  Considerations . 

Accountable  Contract  Management,  AFCMD 

Review  . 

Efforts  to  Strengthen  DOD’s  Quality  Assurance 
Program . 


itt 


.  11 

.  12 
.  13 
.  15 
.  15 
.  16 

.  16 

.  16 _ 

.  16  -esslon  For 

•  16  :s  OR  Aft  if  ~ 
.  17  :  TAB 

inounced 

.  17  ‘-tricot  lon- 


18  _ _ _ 

in  3trlbutlan/ 

Availability  Cod 
lAvall  and/or 
Dlst  Spoclal 


Chapter  Page 

Summary  . 19 

Notes  . 19 

3  RESEARCH  METHODOLOGY . 21 

Description  of  Population  . 21 

Data  Collection  . 22 

Development  and  Validation  of  Survey  Instrument  .  .  24 

Description  of  Survey  Instrument . 25 

Selection  of  Statistical  Tests . 25 

Notes  . 27 

4  FINDINGS . 29 

Demographic  Data:  Program  Directors,  Program 

Managers,  and  Quality  Assurance  Personnel  ....  29 

Summary  of  Demographic  Data . 34 

How  Have  Past  Studies  Addressed  Total 

Quality  Management? . 35 

What  Are  the  Most  Important  Factors 

Affecting  Quality? . 38 

What  Is  the  SPO  Doing  to  Affect  Quality?  . 46 

Effectiveness  of  Quality  Assurance  in  the  SPO  ...  48 

Contract  Incentives  and  Improved  Quality  . 48 

Impact  of  Quality  Problems . 51 

Using  Quality  Improvement . 54 

What  Is  the  Contractor  Doing  to  Affect  Quality?  ....  58 

Are  Indicators  of  Quality  Being  Used? . 59 

Are  SPO  Personnel  Using  Cost  of  Quality  Data 

as  a  Program  Management  Tool?  . 63 

Summary  . 65 

Notes  . 67 

5  SUMMARY  AND  RECOMMENDATIONS . 69 

How  Have  Past  Studies  Addressed  This  Topic?  ....  70 

What  Are  the  Most  Important  Factors 

Affecting  Quality? . 71 

What  Is  the  System  Program  Office  Doing 

to  Affect  Quality?  . 71 

What  Is  the  Contractor  Doing  to  Affect  Quality?  ....  72 

Are  Indicators  of  Quality  Being  Used? . 72 

Are  SPO  Personnel  Using  Cost  of  Quality  Data 

as  a  Program  Management  Tool?  . 73 

Recommendations . 73 

Concluding  Observations  . 76 


iv 


Appendix 


Page 


A  Contractor  Quality  Performance  Evaluation 

Summary .  77 

B  Quality  Award  Fee  Approach  .  83 

C  Letters  . 107 

D  Program  Director  and  Manager  Questionnaire  ....  113 

E  Quality  Assurance  Personnel  Questionnaire  . 125 

F  Program  Director’s  Data . 137 

G  Program  Manager’s  Data  . 147 

H  Quality  Assurance  Military  Personnel  Data . 157 

I  Quality  Assurance  Civilian  Personnel  Data  . 167 

ILLUSTRATIONS 

Figure  Page 

1  Development  Process  that  Designs  Quality 

into  the  Product  .  40 

2  Producibility  Risk  Funding  .  40 

3  Quality — MOST  Important  Objective  .  40 

4  Expending  Sufficient  Resources .  40 

5  Using  Material  Review  Board/Quality 

Deficiency  Reports  .  42 

6  Monitoring  Defects  and  Workmanship  Data .  42 

7  Education  Programs  Aimed  at  Excellence  in 

Management  and  Technical  Fields  .  42 

8  Top  Management  Influencing  the  Quality 

of  Your  Products  .  42 

9  Award-Fee  Contract  Influence  Quality  for 

Your  Products  .  44 


v 


Figure  Page 

10  Firm-Fixed -Price  Contract  Influence  Quality 

of  Your  Products  .  44 

1 1  Frequency  of  Reviewing  Scrap,  Rework,  and 

Repair  Data . 44 

12  Frequency  of  Reviewing  Cost  of  Quality  Data  . 44 

13  Delivery  Schedules  Affected  Quality  of  Your 

Product . 45 

14  Team  Building  between  the  SPO,  CAS,  and 

Contractor  . 45 

1 5  Quality  Improvement  Process  via  Design  Teams . 45 

16  The  Right  Number  of  QA  Personnel  Assigned  to 

the  SPO . 45 

17  Active  Subcontractor  Quality  Improvement 

Process . 47 

18  Subcontractor  Yield  Rates . 47 

19  Contractor  Using  a  Preferred  Vendor  Program 

for  Quality  and  Schedule . 47 

20  Contractor  Emphasis  on  Supplier’s  Quality . 47 

2 1  QA  Personnel’s  Help  in  Achieving  Expected 

Level  of  Quality . 49 

22  Interface  between  Quality  and  Program 

Management  Personnel . 49 

23  Right  Kind  of  QA  Personnel  Assigned  to  the  SPO  ....  49 

24  Staff  QA  Organization  Helps  Achieve  a 

Quality  Product . 49 

25  Headquarters  AFSC  Leadership  Helps  Achieve 

a  Quality  Product . 50 

26  Influence  the  Contractor  Using  Contract 

Requirements . 50 

27  Influence  the  Contractor  Using  Good  Business 

Practices  . 50 


vi 


Figure  f>a£?e 

28  User’s  Influence  on  Contractor’s  Quality  of 

Hardware/Services . 50 

29  Influence  the  Contractor  Using  a  Warranty . 52 

30  Normal  Number  of  Quality  Problems  over 

the  Last  Year . 52 

31  Manufacturing  Errors  Causing  Management 

Problems . 52 

32  Engineering  Errors  Causing  Management 

Problems . 52 

33  Design /Manufacturing  Problems  Effect  on 

Schedule,  Cost,  Performance,  or  Reliability . 53 

34  SPO  Using  Hardware  Quality  Audits  . 55 

35  Contractor  Shares  Your  Quality  Objectives . 55 

36  Measurable  Quality  Improvement  Goals 

Defined  by  Your  Organization . 55 

37  SPO  Monitor  Performance  to  Goals  . 55 

38  Contractor  Performance  Measured  against 

Quality  Standards  . 57 

39  Comparison  of  Planned  versus  Actual 

Work-Hour  Data  . 57 

40  Quality  Personnel  Focus  on  Defect  Prevention 

versus  Detection  . 57 

41  Use  of  Measures  and  Indicators  of  Quality  . 57 

42  Contractor  Using  a  Total  Quality 

Management  Approach . 60 

43  Contractor  Using  Employee  Motivation 

Techniques . 60 

44  Contractor  Instituted  an  Active  Quality 

Improvement  Program  . 60 

45  Contractor  Using  Statistical  Process  Control 

Including  Vendors  . 60 

vil 


Figure  Page 

46  Contractor  Using  Indicators  and  Measures  of 

Quality . 62 

47  Contractor  Affects  Quality — Summary  . 62 

48  Number  of  Significant  Quality  Problems . 62 

49  Using  Field  Performance  Data . 62 

50  Product  Return  Rate  (Cost  to  Rework)  as 

a  Quality  Indicator . 64 

5 1  Degree  of  Manu  facturing  Process 

Standardization  as  an  Indicator  . 64 

52  Production  Yield  Rates  as  a  Quality 

Indicator  . 64 

53  Accept  or  Reject  Rate  at  Work  Centers  as 

a  Quality  Indicator . 64 

54  Review  of  User  Complaint  Data  on  System 

or  Item . 66 

55  Cost  of  Quality  Estimate  as  a  Percentage  of 

Contract  Value . 66 

56  Estimate  of  the  Cost  of  Quality  Components: 

Failure,  Inspection,  and  Prevention  Costs . 66 

57  Confidence  in  Cost  of  Quality  Estimate . 66 

Table 

1  Procurement  Budget  .  4 

2  Number  of  Questionnaires  Mailed . 23 

3  Responses  and  Estimated  Error  Rate . 23 

4  Rank  and  Grade  Frequency  Distribution  . 29 

5  Number  of  Years  in  Program  Management  . 30 

6  Number  of  Years  in  Quality  Assurance  . 30 

7  Number  of  Systems  or  Items  Managed  . 31 

vlil 


Table 


8  Number  of  Years  on  Current  Program . 

9  Acquisition  Phase . 

10  Full-time  Quality  Assurance  Personnel  Assigned  .  .  .  . 

1 1  Training  or  Education  on  How  to  Improve  Quality  .  .  .  . 

12  Hours  Spent  on  Quality  Issues  . 

13  Contributors  to  Better  Quality  . 

14  Detractors  from  Better  Quality  . 


Page 

32 

32 

33 

33 

34 
38 
43 


v 


J 


Foreword 


Production  inefficiencies  in  many  defense  industries  have  resulted  in 
significant  cost  growth  for  major  weapon  systems.  These  costs  have  not 
gone  unnoticed.  The  Department  of  Defense  (DOD)  has  implemented  many 
programs  in  an  attempt  to  reduce  these  inefficiencies,  but  these  efforts  have 
met  with  little  apparent  success.  More  recently  the  Defense  Department 
has  adopted  a  philosophy — total  quality  management — that  could  finally 
lower  these  costs.  This  approach  will  require  a  reeducation  and  cultural 
change  of  both  the  DOD  and  contractor  work  forces. 

In  this  study.  Maj  John  A.  Campbell  uses  a  survey  questionnaire  to 
determine  to  what  degree  the  Air  Force  Systems  Command  (AFSC)  acquisi¬ 
tion  work  force  understands  what  is  causing  these  production  inefficiencies 
and  if  it  is  applying  management  techniques  that  can  reduce  the  cost  of 
these  inefficiencies.  He  surveys  program  directors,  program  managers,  and 
quality  assurance  personnel  at  AFSC’s  five  major  product  divisions. 

The  most  significant  result  of  this  study  is  that  it  establishes  an  important 
baseline  that  we  can  use  to  measure  our  commitment  to  addressing  these 
production  inefficiencies.  The  data  strongly  supports  the  notion  that  the 
total  quality  management  philosophy  using  continuous  improvement  is  not 
well  understood  by  the  acquisition  work  force.  The  data  also  indicates  that 
the  work  force  does  not  understand  the  magnitude  of  these  production 
inefficiencies.  Finally,  a  communication  problem  in  the  program  offices 
inhibits  the  work  force  from  applying  those  management  techniques  that 
can  minimize  these  inefficiencies.  ^ 


DENNIS  D.  DONEEN 
Brigadier  General.  USAF 
DCS/Product  Assurance  and 
Acquisition  Logistics 
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Preface 


My  charter  from  Col  Robert  J.  Pratt,  director,  product  assurance  en¬ 
gineering  in  the  Office  of  the  Deputy  Chief  of  Staff.  Product  Assurance  and 
Acquisition  Logistics.  Headquarters  Air  Force  Systems  Command,  was  to 
determine  how  well  the  acquisition  work  force  understands  and  is  using 
total  quality  management  (TQM).  Over  the  last  two  decades  some  20  major 
studies  have  made  hundreds  of  recommendations  on  how  to  improve  the 
quality  assurance  function.  Many  of  the  recommendations  have  been 
implemented,  but  critics  still  say  the  quality  of  Defense  Department 
products  is  not  as  good  or  as  cost  effective  as  it  should  be. 

I  have  tried  to  structure  this  document  for  two  categories  of  readers:  the 
senior  management  personnel  interested  in  key  issues,  findings,  and 
recommendations  and  the  acquisition  personnel  interested  in  improving 
the  quality  and  reducing  the  cost  of  our  systems.  The  senior  managers 
should  read  chapter  1  (Introduction),  chapter  2  (Quality  Assurance:  A 
Survey  of  Previous  Studies,  the  first  two  sections.  Introduction  and  Over¬ 
view),  and  chapter  5  (Summary  and  Recommendations).  The  rest  of  the 
acquisition  work  force  should  read  all  five  chapters.  Chapter  1  identifies 
the  problem.  Chapter  2  reviews  six  major  quality  studies  conducted  during 
the  last  10  years  and  provides  the  background  for  this  study.  Chapter  3 
describes  the  research  methodology  used  in  this  study,  and  chapter  4 
presents  the  survey  data  gathered  during  this  study.  Chapter  5  outlines 
the  conclusions  and  proposes  recommendations  to  improve  quality  and 
productivity  in  major  system  acquisitions. 

I  cannot  mention  all  of  the  individuals  who  aided  and  encouraged  me  in 
the  course  of  this  project.  1  would  like  to  thank  Maj  Gen  David  J.  Teal  for 
the  interest  and  support  he  provided  in  this  study.  I  would  like  to  especially 
thank  all  the  senior  program  directors,  program  managers,  and  quality 
assurance  personnel  who  took  the  time  to  complete  my  survey  question¬ 
naire.  1  owe  special  thanks  to  Lt  Col  Robert  Bienvenue,  Maj  George  Noyes, 
and  Grover  Cleveland  for  all  the  time  and  effort  they  most  readily  gave  in 
support  of  this  study. 

The  people  at  the  Air  University  Center  for  Aerospace  Doctrine.  Research, 
and  Education  have  been  superb.  A  special  thanks  goes  to  Lt  Col  Manfred 
Koezur.  who  always  went  beyond  his  duties  to  ensure  that  the  researchers 
had  everything  required  to  do  the  research.  This  study  would  not  have  been 
possible  without  his  assistance  in  getting  supplies,  computers,  and  plotters. 
Another  special  thanks  goes  to  my  committee  chairman.  Dr  Stephen  Blank, 
who  provided  the  right  combination  of  guidance  and  freedom  to  allow  me 
to  perform  this  study.  I  cannot  thank  Thomas  C.  Ijobenstein  enough  for 
going  beyond  his  editorial  duties  to  help  structure  and  ensure  the 
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readability  of  this  document.  I  would  also  like  to  thank  that  small  group  of 
other  command -sponsored  research  fellows  who  were  always  there  provid¬ 
ing  the  support  and  encouragement  needed  to  complete  this  project.  I 
would  be  remiss  if  I  did  not  thank  Jesse  Barron  and  Judy  Lacour,  1973d 
Communications  Group,  for  all  the  help  they  gave  in  writing  the  interface 
with  the  Statistical  Package  for  the  Social  Sciences-X  (SPSS-X)  programs. 
Their  assistance  was  invaluable  for  this  study. 

Finally,  I  must  acknowledge  the  special  contributions  of  my  family.  My 
wife  Pat’s  love  and  support  long  distance  from  Ohio  make  her  the  most 
significant  contributor  to  this  project.  I  could  never  adequately  express  in 
v  i  s  my  appreciation  for  her  dedicated  support.  I  must  give  a  special 
thanks  to  my  son  Keith  who  understood  why  dad  had  to  go  to  Alabama  for 
a  year  and  miss  some  of  the  important  moments  in  his  teenage  life.  I  will 
always  be  grateful  to  my  family  and  special  friends  for  their  love,  support, 
and  encouragement. 


</  JOHN  A.  CAMPBELL.  Maj,  USAF 
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CHAPTER  1 


Introduction 


During  the  last  several  years  the  press.  Congress,  and  even  people  inside 
the  Department  of  Defense  (DOD)  have  strongly  criticized  the  way  the  armed 
services  develop  and  acquire  new  weapon  systems  and  related  hardware. 
Much  of  this  criticism  has  focused  on  highly  publicized  examples  of 
mismanagement  and  waste  such  as  $700  toilet  seats  and  $7,000  coffee 
makers.  The  Defense  Department,  in  recognizing  these  problems,  has 
applied  much  energy  and  resources  to  solving  any  problems  that  exist.  As 
a  result,  charges  of  the  Defense  Department  paying  exorbitant  prices  are 
now  rare.  However,  such  articles  have  been  replaced  with  highly  critical 
ones  focusing  on  cost  growth,  late  deliveries,  and  unreliable  and  poor 
quality  systems.  Examples  include  several  months’  delay  in  the  delivery  of 
the  advance  technology  bomber  (B-2), 1  poor  reliability  of  the  Maverick  and 
Phoenix  missiles,  and  delivery  of  the  newest  intercontinental  ballistic 
missile  (ICBM),  the  Peacekeeper  (MX),  with  poor  quality  parts  in  itsguidance 
system.  In  addition  to  the  press,  some  very  high  ranking  DOD  officials  have 
been  critical  of  how  the  armed  services  are  managing  weapon  acquisition. 

In  a  1982  Defense  Logistics  Agency  productivity  seminar,  Gen  Robert  T. 
Marsh,  then  commander  of  Air  Force  Systems  Command  (AFSC),  said  that 
US  defense  industries  needed  a  “quality  revolution."  He  estimated  that 
AFSC  alone  was  wasting  “a  minimum  of  $570  million  a  year  on  scrap, 
rework  and  repair  at  the  prime  contractor’s  plant."2  He  further  estimated 
that  these  “hidden  factory"  costs  have  increased  60  percent  since  1976. 
That  same  year  Sen  John  W.  Warner  (R-Va.),  speaking  at  a  quality  and 
productivity  conference,  estimated  these  hidden  factory  costs  to  be  as  high 
as  15  percent  of  the  defense  procurement  budget  and  announced  he  was 
planning  to  sponsor  hearings  to  investigate  these  costs.3 

Robert  B.  Costello,  under  secretary  of  defense  for  acquisition,  stated  in 
February  1988  that  because  of  red  tape  and  production  inefficiencies  the 
government  was  losing  as  much  as  $45  billion  “of  the  yearly  military 
purchasing  budget."4  TTiese  production  inefficiencies  are  the  same  hidden 
factory  costs  referred  to  by  General  Marsh  and  Senator  Warner  in  1982. 
Costello  went  on  to  say  that  this  loss  amounted  to  about  20  to  30  percent 
of  the  $  1 50-billion  total  procurement  budget  and  was  not  adding  any  value 
to  the  weapon  system.'  Further  emphasizing  a  need  for  improvement,  a 
poll  reported  by  the  AIR  FORCE  Magazine  stated  that  “a  majority  of  citizens 
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believes  Lhai  the  govenun  is  being  cheated  It'll  anci  right  in  military 
procurements."6  This  lack  ol  faith  by  US  taxpayers  can  have  a  serious 
impact  on  how  the  Defense  i)cpartment  maintains  combat  readiness 
through  acquiring  new  weapon  systems. 

The  Defense  Department  has  the  responsibility  and  obligation  to  the 
taxpayer  to  acquire  only  those  systems  required  lor  national  security,  and 
in  the  most  cost-effective  and  efficient  manner  possible.  But,  if  DOD 
continues  not  to  address  t  ht  se  hidden  factory  costs.  Congress  will  focus  its 
attention  on  that  problem,  as  it  has  in  many  other  instances,  and  will  be 
justified  in  providing  a  solution  to  it.  Congress  not  only  responds  to  obvious 
problems  of  mismanagement  and  waste  but  to  the  concerns  of  its  con¬ 
stituents.  Hence,  in  the  last  few  years  Congress  has  become  increasingly 
involved  in  overseeing  the  management  of  DOD  weapon  systems.  This 
oversight  may  come  in  the  form  of  reduced  budgets,  new  legislation  to  fix 
the  problem,  or  increased  “micromanagement,"  namely,  requirements  to 
report  directly  to  Congress.  Many  in  DOD  debate  the  value  of  some  of  t  hese 
congressionally  imposed  fixes  because  they  require  additional  reporting  to 
Congress.  This  reporting  requirement  has  significantly  increased  the  work 
load  for  what  many  say  is  an  already  overburdened  acquisition  work  force. 
A  recent  example  of  congressional  concern  with  DOD  program  management 
was  the  authorizing  of  $10  million  to  be  used  specifically  to  develop  a  cost 
control  management  program  for  the  advanced  technology  bomber  (B-2).7 
This  move  was  a  result  of  many  articles  and  reports  of  significant  cost 
overruns  on  the  program. 

The  Competition  in  Contracting  Act  of  1984  (PL  98-369)  was  another 
example  of  Congress  trying  to  ensure  that  the  Department  of  Defense 
introduces  more  competition  into  its  procurement  practices.  For  many 
years  Congress  had  felt  that  the  Defense  Department  and  other  federal 
agencies  had  not  been  awarding  contracts  in  a  fully  competitive  environ¬ 
ment.  Basic  economic  theory  says  that  in  a  free-market  environment, 
competition  will  reduce  the  price  of  an  item.  As  a  result  of  this  new  law 
more  than  one-third  of  the  Federal  Acquisition  Regulation  (FAR),  which  was 
only  one  year  old,  was  revised.  FAR,  which  became  law  in  1983,  con¬ 
solidated  the  procurement  procedures  of  all  federal  agencies  into  one 
regulation;  it  specified  how  all  federal  agencies  would  procure  items.  This 
consolidating  of  procurement  procedures  has  had  tremendous  impact  on 
many  federal  agencies  and  contractors. 

The  DOD  Supplement  to  the  Federal  Acquisition  Regulation  was  yet 
another  step  by  Congress  to  try  to  make  the  suppliers  of  major  weapon 
systems  more  responsible  for  the  products  they  sell  the  government .  Many 
in  Congress  believed  that  contractors  were  not  providing  as  reliable  a 
product  as  possible.  Congressional  critics  thought  that  DOD  contractors 
should  warrant  their  products  similar  to  the  way  many  commercial 
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products  are  warranted.  Thus,  DOD  FAR  Supplement,  Part  46.770-2, 
Policy,  stated: 

Unless  waived  under  46.770-9.  after  1  January  1985.  the  Military  Departments  and 
Defense  Agencies  may  not  enter  into  a  contract  for  the  production  of  a  weapon 
system  .  .  .  unless: 

(1)  a  prime  contractor  for  the  weapon  system  provides  the  United  States  with  written 
warranties. 

DOD  FAR  Supplement,  Part  46.770-9.  required  that  a  copy  of  any  waiver  be 
submitted  to  the  Senate  and  House  Committees  on  Armed  Services  and  on 
Appropriations.  This  requirement  was  an  obvious  example  of  Congress 
becoming  more  involved  in  how  DOD  manages  its  weapon  system  acquisi¬ 
tions  and,  in  so  doing,  increasing  the  burden  of  reporting. 

In  the  1986  DOD  Authorization  Act  (PL  99-145),  Congress  stipulated  that 
certain  acquisition  personnel  have  specific  training  and  education.  Section 
1624  required  mandatory  training  for  all  personnel  responsible  for  assuring 
quality  in  contractor  facilities.  Again,  Congress  believed  that  many  of  the 
quality  assurance  personnel  responsible  for  accepting  DOD  products  had 
not  been  adequately  educated  or  trained  to  perform  this  function. 

Reducing  hidden  factory  costs  and  increasing  the  skills  of  acquisition 
personnel  in  quality  improvement  became  even  more  important  in  view  of 
recent  trends  in  the  defense  budgets.  During  the  early  1980s,  the  Defense 
Department  was  able  to  obtain  increasing  budgets  and  made  significant 
strides  in  increasing  the  capability  of  the  armed  forces.  However,  the  trend 
for  the  future  indicated  a  decreasing  procurement  budget  (table  1).  If  this 
downward  trend  were  to  continue,  as  it  did  in  the  late  1970s,  the  state  of 
readiness  of  our  military  forces  could  again  be  seriously  threatened.  In 
1980  the  General  Accounting  Office  (GAO)  reported  to  Congress  that  the 
rising  cost  of  weapon  systems  resulted  in  the  acquisition  of  fewer  units  of 
equipment  than  needed  by  the  armed  services,  which  affected  their  combat 
readiness.  There  was  little  doubt  that  the  cost  of  weapon  systems  would 
continue  to  increase.  Given  the  shrinking  procurement  budget  as  indicated 
using  costs  of  weapons,  the  Defense  Department  should  expect  lean  budget 
years  ahead.  Therefore,  DOD  must  become  more  effective  and  efficient  in 
procuring  material.  Not  to  do  so  could  have  serious  consequences  on  our 
national  security. 

This  research  project  evaluated  how  well  Air  Force  Systems  Command 
quality  assurance  (QA)  personnel,  program  directors  (PD),  and  program  and 
project  managers  (PM)  understand  what  hidden  factory  costs  are  and 
determined  whether  those  workers  were  performing  the  necessary  functions 
to  reduce  them.  When  General  Marsh  discussed  hidden  factory  costs,  he 
was  addressing  primarily  scrap,  rework,  and  repair  costs.  These  three 
factors  served  as  primary  indicators  of  a  product’s  quality8  and  were  the 
major  focus  of  this  research  project. 


3 


TABLE  1 


Procurement  Budget 
(Millions) 


Department  of  Defense 
Air  Force 

Air  Force  Systems  Command 


FY  86  FY  87  FY  88 

$92,506  $80,234  $81 ,027a 

37,100  34,893  32,534b 

19,129  13,557  11,174c 


a.  Annual  Report  to  the  Congress,  Fiscal  Year  1989  (Washington,  D.C.:  Government  Printing 
Office,  1 1  February  1988),  297. 

b.  Defense  Budgets  (Washington,  D.C.:  Government  Printing  Office,  1 1  December  1 987),  F-i. 

c.  Rita  Maldonado,  budget  analyst,  Budget  Management  Division,  Deputy  Chief  of  Staff.  Comp¬ 
troller,  Headquarters  Air  Force  Systems  Command,  Andrews  AFB,  Md.,  telephone  interview  with 
author,  1 0  February  1 988. 


Quality  Regulations 


Four  primary  documents  define  and  regulate  what  actions  the  Air  Force 
will  take  to  assure  quality  in  its  acquisitions.  These  documents  are  the 
Federal  Acquisition  Regulation,  Part  46.105,  Contractor  Responsibilities; 
the  DOD  Supplement  to  FAR,  Part  46.102,  Policy,  DOD  Directive  4155.1, 
Quality  Program ;  and  AFR  74- 1 ,  Quality  Assurance  Program. 

Federal  Acquisition  Regulation 

The  Federal  Acquisition  Regulation  sets  the  policy  and  requirements  for 
quality  assurance  in  government  procurements.  FAR,  Part  46. 105,  states: 

(a)  The  contractor  is  responsible  for  carrying  out  its  obligations  under  the  contract 

by— 

(1)  Controlling  the  quality  of  supplies  or  services:  (and) 

(2)  Tendering  to  the  Government  for  acceptance  only  those  supplies  or  services 
that  conform  to  contract  requirements. 

DOD  Supplement  to  the  Federal  Acquisition  Regulation 

The  DOD  Supplement  io  FAR  identifies  specific  requirements  expected  of 
the  Defense  Department  concerning  the  contractor’s  responsibility  for 
delivering  quality  products.  Part  46.102  (72)(2)  states: 

The  Government  shall  hold  contractors  responsible  for  the  quality  of  products  and 
services  by  means  of: 

(i)  contract  provisions  that  place  responsibility  on  contractors: 

(II)  the  Government's  exercising  Its  right  to  reject  or  return  contractor- responsible 
defective  i.  ms  for  repair,  correction  or  replacement. 
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DOD  policy  is  that  the  manufacturer  has  ultimate  responsibility  for  produc¬ 
ing  quality  products.  This  policy  is  set  forth  in  DODD  4155. 1  and  applies 
to  all  DOD  components. 

DODD  4155.1,  Quality  Program 

DODD  4155.1  requires  all  DOD  components  to  ensure  that  all  services 
and  products  acquired  conform  to  specified  requirements.  DOD’s  stated 
policy  in  (his  directive  repeats  the  requirements  of  DOD  FAR  Supplement, 
Part  46. 102  (72){2):  the  “contractor  shall  be  held  responsible  for  the  quality 
of  products  and  services."  However,  this  directive  goes  on  to  say  that 
contractors  are  to  be  held  responsible  for  the  quality  of  their  suppliers  and 
vendors.  It  further  states  that  DOD  components  will  not  award  contracts 
to  contractors  who  have  a  history  of  delivering  poor  quality  products  or 
services.  This  directive  also  places  requirements  on  DOD  components  to 
maintain  a  quality  history  on  the  contractor  and  a  product  deficiency 
reporting  system  that  can  be  used  to  evaluate  a  contractor.  Additionally, 
it  makes  the  program  (system)  manager  responsible  and  accountable  for 
assuring  that  products  or  services  delivered  to  the  Department  of  Defense 
are  of  acceptable  quality.  One  final  important  requirement  of  this  directive 
is  that  it  requires  the  program  (system)  manager  to  quantify  quality 
characteristics  whenever  possible.9 

AFR  74-1,  Quality  Assurance  Program 

The  Air  Force  implements  this  DOD  policy  through  AFR  74-1.  This 
regulation  specifies  the  objectives  of  the  Air  Force  quality  assurance  pro¬ 
gram.  Some  of  the  Air  Force's  objectives  include  making  sure  that  all 
products  and  services  conform  to  specified  quality  requirements.  These 
specified  requirements  must  be  practical,  enforceable,  and  necessary. 
Quality  deficiencies  that  affect  the  mission  and  user  satisfaction  must  be 
prevented  and  action  must  be  taken  to  eliminate  the  causes  of  these 
deficiencies.  Air  Force  policy  emphasizes  preventing  quality  deficiencies 
and  recurrence  of  those  conditions  which  cause  deficiencies.  Additionally, 
data  from  the  users  of  the  products  and  services  acquired  will  be  analyzed 
to  assess  and  improve  the  quality  assurance  program.  The  regulation  also 
requires  that  QA  people  participate  in  design  reviews  throughout  all  ac¬ 
quisition  phases. 

On  major  programs,  quality  assurance  personnel  are  to  perform  inde¬ 
pendent  quality  assessments  throughout  the  system’s  life  cycle.  This 
process  will  be  an  objective  evaluation  of  program  adequacy  from  a  quality 
viewpoint  to  verify  that  quality  characteristics  are  quantified  and,  when 
possible,  specified  and  designed  into  the  product.  The  directives  also  list 
functional  requirements  for  the  contracting  organization,  contracting  ad¬ 
ministration  organization  (CAO),  and  the  major  commands  (MAJCOMs). 
The  contracting  organization  must  ensure  that  contracts  include  inspection 
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and  acceptance  provisions;  it  must  maintain  data  concerning  unsatisfac¬ 
tory  products;  and  it  will  not  award  contracts  to  contractors  who  deliver 
poor  quality  products.  The  CAO  must  ensure  that  the  contractor  complies 
with  the  contract  quality  before  accepting  the  product,  must  maintain 
historical  information  on  the  contractor’s  quality  performance,  and  may 
reduce  surveillance  when  the  contractor’s  quality  performance  justifies 
doing  so.  The  MAJCOM  will  plan,  program,  and  budget  for  adequate 
manpower  resources  to  ensure  an  effective  QA  program,  evaluate  the  quality 
of  products  at  regular  intervals,  and  ensure  that  QA  personnel  are  properly 
trained  and  motivated.  Finally,  the  regulation  identifies  QA  tasks  to  be 
performed  during  each  acquisition  phase.10 

These  regulations  are  the  primary  documents  that  establish  policy  and 
provide  guidance  to  the  DOD  and  Air  Force  on  how  to  procure  quality 
products.  By  law  (FAR),  the  contractor  must  deliver  a  product  that  meets 
the  quality  requirements  specified  in  the  contract.  Contrary  to  the  results 
of  the  opinion  poll  reported  by  AIR  FORCE  Magazine,  contractors  are  not 
out  to  cheat  DOD.  Alan  C.  Chase,  a  professional  staff  member  of  the  House 
Armed  Services  Committee,  says  that  these  impressions  (cheating  DOD)  are 
driven  more  by  belief  than  by  fact.  He  also  said  that  he  "has  not  found 
reason  to  question  the  integrity  of  the  defense  industry  as  a  whole.”1 1 

Much  of  the  problem  arises  from  the  difficulty  that  the  work  force  has  in 
specifying  measurable  parameters  of  quality  for  complex  state-of-the-art 
equipment.  A  short  story  written  by  John  Guaspari.  titled  I  Know  It  When 
I  See  It:  A  Modem  Fable  about  Quality,  clearly  demonstrates  this  point.  The 
premise  of  this  story  is  that  sometimes  quality  is  difficult  to  put  into  words 
but  the  buyers  know  quality  when  they  see  it.  This  book  discusses  a 
fictional  company  that  made  paper  products.  The  company  begins  to  lose 
customers  even  though  its  product  met  their  specifications.  The  point  of 
the  story  is  that  the  customer  has  needs  that  must  be  met.  and  those  are 
not  necessarily  the  same  as  the  specifications.12  A  recently  published 
handbook  by  Headquarters  AFSC  identifies  many  management  indicators 
that  will  be  very  helpful  to  the  work  force  in  assessing  the  health  of  a 
program.13 

AFR  74-1  defines  quality  as:  “The  composite  of  material  attributes 
including  performance;  features  and  characteristics  of  a  product  or  service 
to  satisfy  a  given  need.”  The  idea  of  satisfying  the  need  parallels  the  thesis 
of  Guaspari’s  story.  Although  it  is  important  to  identify  specific  require¬ 
ments  for  quality  to  the  contractor,  the  final  factor  must  be  how  well  the 
product  satisfies  its  intended  use.  This  fitness  for  use  is  becoming  the 
working  definition  of  a  quality  product  for  many  acquisition  personnel  in 
AFSC.  Most  products  and  services  used  by  the  Air  Force  are  acquired  by 
Air  Force  Systems  Command  (AFSC)  and  Air  Force  Logistics  Command 
(AFLC).  This  research  focuses  on  how  AFSC  ensures  that  the  Air  Force 
receives  a  quality  product . 
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Mission  of  Air  Force  Systems  Command 


The  Air  Force  Systems  Command  is  responsible  for  acquiring  most 
weapon  and  related  systems  in  the  Air  Force  inventory  today.  Specifically. 
AFSC  has  three  primary  responsibilities:  to  advance  aerospace  science  and 
technology:  to  adapt  these  advances  into  developing  and  improving  opera¬ 
tional  systems;  and  to  acquire  qualitatively  superior,  logisticallv  supported 
aerospace  systems  at  the  most  effective  cost.*4  AFSC  controls  more  than 
one-third  of  the  total  Air  Force  budget.  Most  of  AFSC’s  share  of  this  budget 
goes  toward  research,  development,  testing,  and  hardware  procurement. 
Systems  Command  continuously  seeks  better,  more  efficient  techniques  of 
managing  the  resources  devoted  to  national  defense. 

To  accomplish  its  mission.  Air  Force  Systems  Command  is  organized  into 
buying  (product)  divisions,  test  centers,  a  contract  management  division, 
and  ot  hers.  This  study  focuses  on  quality  assurance  at  (he  buying  divisions 
but  must  discuss  the  mission  of  the  Contract  Management  Division. 
Kirtland  AFB.  New  Mexico,  because  of  its  responsibility  for  quality  as¬ 
surance.  The  product  divisions  are  the  Armament  Division  (AD).  Eglin  AFB. 
Florida:  the  Aeronautical  Systems  Division  (ASD),  Wright-Patterson  AFB. 
Ohio:  the  Ballistic  Missile  Office  (BMO),  Norton  AFB.  California:  the 
Electronic  Systems  Division  (ESD).  Hanscom  AFB.  Massachusetts:  and  the 
Space  Division  (SD),  Los  Angeles  AFS,  California.  The  buying  divisions  are 
organized  into  program  offices  where  in  most  cases  the  quality  assurance 
personnel  are  integrated  into  that  organization. 

Air  Force  Contract  Management  Division 

The  Air  Force  Contract  Management  Division  (AFCMD)  is  the  primary  Air 
Force  agency  responsible  for  contract  management  functions  at  contractor 
plants  assigned  to  the  Air  Force  by  DOD.  The  division  performs  these 
functions  for  government  program  managers  and  buying  agencies  and 
evaluates  the  contractor’s  management  systems,  practices,  and  contract 
performance.15 

AFCMD's  4,000  military  and  civilian  personnel  perform  contract  manage¬ 
ment  at  24  plants  throughout  the  United  Stales.  These  Air  Force  plant 
representative  offices  (AFPROs)  have  an  average  of  about  155  personnel 
assigned,  but  this  number  ranges  from  approximately  300  at  large  plants 
to  60  at  small  plants.  About  39  percent  of  these  AFCMD  personnel  are 
responsible  for  the  quality  assurance  function,  including  inspection. 16  The 
other  personnel  provide  contract  management  services  in  manufacturing 
operations,  subcontract  management,  Industrial  materiel  management, 
contract  payment,  and  engineering  and  program  support.17  Until  just 
recently.  AFCMD  primarily  used  the  contractor  management  system 
evaluation  program  (CMSEP)  to  evaluate  the  contractor’s  performance  in 
the  above  mentioned  areas.  CMSEP  used  307  questions  to  evaluate  the 
adequacy  of  and  compliance  with  the  contractor's  documented  manage - 
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ment  systems.  The  largest  number  of  questions  (67)  were  dedicated  to  the 
quality  assurance  function.  The  Air  Force’s  philosophy  of  using  CMSEP  for 
contract  management  was  that  if  the  contractors’  systems  are  adequate  and 
comply  with  their  own  procedures,  then  the  product  should  be  good.  The 
results  of  these  evaluations  were  provided  to  the  government  program 
managers  and  buying  agencies.  However,  many  AFCMD  officials  have 
questioned  the  effectiveness  of  this  system  for  several  years,18  and  the 
division  has  recently  decided  to  discontinue  using  CMSEP  as  its  primary 
contractor  performance  tool. 

Armament  Division 

The  Armament  Division  conducts  planning,  research,  development,  test, 
evaluation,  and  initial  acquisition  of  nonnuclear  armament.  It  also  con¬ 
ducts  development  and  acquisition  for  improved  range  and  instrumental 
systems.  The  division  is  the  center  of  expertise  for  conducting  tests  for  all 
acquisition  phases  on  such  systems  as  nonnuclear  munitions,  armament 
avionics,  and  radiating  systems  and  for  conducting  tests  of  flightworthi- 
ness.19  The  Armament  Division  has  a  total  of  13  military  and  civilians 
assigned  to  perform  the  quality  assurance  function. 

Aeronautical  Systems  Division 

The  Aeronautical  Systems  Division  plans  and  manages  the  development 
and  acquisition  of  aeronautical  systems,  subsystems,  and  equipment, 
including  in-flight  test  and  evaluation.  The  division  provides  airborne 
test-beds  and  airborne  range  instrumentation  services  to  support  AFSC 
organizations,  DOD  agencies,  the  National  Aeronautics  and  Space  Ad¬ 
ministration  (NASA),  and  the  Federal  Aviation  Administration.20  ASD  has 
66  military  and  civilian  personnel  assigned  to  the  quality  assurance  func¬ 
tion. 

Ballistic  Missile  Office 

The  Ballistic  Missile  Office  plans,  implements,  and  manages  programs  to 
acquire  ballistic  missile  systems,  subsystems,  and  equipment.  BMO  also 
manages  the  alteration  of  missile  sites  and  launch  facilities.21  BMO  has 
12  military  and  civilian  employees  assigned  to  the  quality  assurance 
function. 

Electronic  Systems  Division 

The  Electronic  Systems  Division  plans,  manages,  and  conducts  tech¬ 
nological  development  (including  research  and  exploratory,  advanced,  and 
engineering  development),  acquisition,  logistics  support  planning,  installa¬ 
tion,  and  delivery  of  command,  control,  communication,  and  intelligence 
(C3I)  systems  and  ground  electronic  systems.22  ESD  has  approximately  60 
military  and  civilian  employees  assigned  to  the  quality  assurance  function. 
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Space  Division 

The  Space  Division  plans,  programs,  and  manages  projects  to  acquire 
space  systems,  subsystems,  and  equipment.  The  division  provides  for  the 
maintenance,  construction,  or  alteration  of  launch,  tracking,  and  support 
facilities;  conducts  launch  and  flight-test  and  evaluation  support;  and 
performs  launch,  on-orbit  satellite  tracking,  data  acquisition,  test  and 
evaluation,  and  command  and  control  of  DOD  satellites.23  The  Space 
Division  has  43  military  and  civilian  employees  assigned  to  perform  the 
quality  assurance  function  at  Los  Angeles  AFS. 


Research  Objective 

As  noted  earlier,  AFSC  has  always  sought  more  efficient  and  effective 
methods  to  acquire  weapon  systems.  AFSC  has  sponsored  more  than  a 
dozen  research  projects  over  the  years,  many  of  which  have  focused  on 
improving  the  quality  assurance  function.  These  studies  have  led  to  many 
significant  changes  to  the  way  quality  assurance  is  performed  in  AFSC.  The 
primary  reason  for  doing  this  research  is  to  see  if  AFSC  acquisition 
personnel  are  focusing  their  attention  properly  on  the  hidden  factory  costs 
described  earlier  by  General  Marsh. 

More  specifically,  the  objective  of  this  research  project  is  to  determine 
how  well  AFSC  acquisition  personnel  are  performing  those  tasks  that  can 
effectively  control  and  reduce  those  hidden  factory  costs.  Although  these 
costs  exist  throughout  DOD,  this  research  project  is  limited  to  AFSC.  This 
research  problem  consists  of  finding  answers  to  six  basic  research  ques¬ 
tions. 

1 .  What  research  projects  and  studies  addressing  product  quality  have 
been  performed  in  the  past? 

2.  What  are  the  most  important  factors  affecting  productivity  and 
quality? 

3.  What  is  the  SPO  doing  to  affect  quality? 

4.  What  is  the  contractor  doing  to  affect  quality? 

5.  To  what  extent  are  indicators  of  quality  used? 

6.  To  what  extent  are  SPO  personnel  using  the  cost  of  quality  as  a 
program  management  concept? 

The  answer  to  these  questions  will  provide  AFSC  an  indication  of  how  well 
the  command  is  addressing  hidden  factory  costs. 
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CHAPTER  2 


Quality  Assurance: 

A  Survey  of  Previous  Studies 


Approximately  30  studies,  dated  back  to  1962,  have  addressed  problems 
in  the  Department  of  Defense  (DOD)  quality  assurance  programs.  Six 
recent  research  studies  have  direct  application  to  how  the  Air  Force  Systems 
Command  (AFSC)  performs  the  quality  assurance  function.  These  studies 
provide  the  necessary  background  to  develop  the  scope  and  methodology 
for  the  current  study.  Headquarters  AFSC  commissioned  these  six  studies, 
conducted  during  the  last  10  years,  to  seek  solutions  to  problems  the 
command  was  experiencing  in  the  quality  of  the  products  it  was  acquiring. 

A  comprehensive  1977  study  set  the  standard  research  methodology  for 
the  studies  that  followed.  Each  study  team  reviewed  findings  and  recom¬ 
mendations  of  previous  studies  and  reviewed  DOD  and  Air  Force  policies 
and  regulations  concerning  quality.  Most  study  groups  conducted  inter¬ 
views  with  high-level  acquisition  personnel  in  the  DOD.  the  service  depart¬ 
ments  and  headquarters,  the  service  acquisition  headquarters,  contract 
administration  organizations,  contractors,  and  professional  organizations. 
The  study  teams  gathered  data  using  surveys  administered  to  various 
acquisition  personnel.  The  average  study  lasted  six  to  12  months  and 
involved  approximately  10  personnel  from  many  acquisition  backgrounds, 
including  several  general  officers. 

These  studies  made  many  important  recommendations  and  led  to  several 
changes  in  how  the  Air  Force  performs  its  quality  functions.  One  significant 
change  recommended  by  the  Quality  ’77  1  (1977)  and  Quality  Horizons2 
(1979)  studies  was  creating  the  office  of  Deputy  Chief  of  Staff.  Product 
Assurance  and  Acquisition  Logistics  (HQ  AFSC/PL),  to  oversee  the  quality 
function.  This  action  elevated  the  level  of  importance  of  the  quality  as¬ 
surance  function.  Another  recommendation  by  these  studies  resulted  in 
significantly  increasing  the  size  of  the  QA  work  force  at  the  five  product 
divisions.  The  Accountable  Contract  Management,  AFCMD  Review  (1984). 
recommended  that  at  least  100  personnel  were  needed  in  the  buying 
divisions  to  ensure  that  quality  and  producibility  received  its  proper  atten¬ 
tion  in  the  design  phase. 

Each  of  these  studies  identified  a  lack  of  training  for  the  QA  personnel 
as  a  major  deficiency.  They  recommended  mandatory  training  programs, 
specific  training  courses  for  the  Air  Force  Contract  Management  Division 
(AFCMD)  personnel,  intern  programs,  and  on-the-job-training  (OJT) 
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programs.  These  three  studies  and  two  others.  Contract  Incentives  for 
Product  Quality  (1980)4  and  Project  Quality  (1982). 5  recommended  that  the 
Air  Force  use  more  contract  incentives  to  motivate  the  contractor  to  deliver 
better  quality  products.  Four  of  the  studies — Quality  '77 ,  Quality  Horizons, 
Project  Quality,  and  Accountable  Contract  Management,  AFCMD  Review — 
also  concluded  that  AFSC  needed  a  policy  or  regulation  that  clearly  specifies 
that  quality  has  a  high  level  of  importance  in  the  acquisition  process.  The 
current  commander  of  AFSC.  Gen  Bernard  Randolph,  recently  issued  a 
policy  statement*’  and  a  letter7  stressing  the  priority  of  quality  in  AFSC. 

three  studies — Quality  '77,  Quality  Horizons,  and  Quality  Assurance: 
efforts  to  Strengthen  DOD’s  Program  (1986)8 — specifically  recognized  the 
need  for  (he  Air  Force  to  enforce  its  quality  requirements  and  make 
contractors  more  responsible  for  the  quality  of  their  products.  Further¬ 
more.  all  six  studies  undertaken  during  this  10-year  period  identified  the 
need  for  the  government  to  use  indicators  of  quality  data  to  assess  and 
manage  contractors.  The  Government  Accounting  Office  (GAO)  in  its  1986 
study.  Quality  Assurance:  Efforts  to  Strengthen  DOD's  Program,  was  very 
critical  of  the  Defense  Department's  and  the  Air  Force's  lack  of  use  of  this 
data.9 


Quality  *77 

Maj  Gin  .tames  W.  Stansberry .  the  AFSC  deputy  chief  of  staff  for 
■  n  ni!'..'iit  and  nianuiactui  ing  (AFSC/1T),  commissioned  the  Quality  '77 
iai  iy  in  March  1977.  He  expressed  several  concerns.  Among  others  he 
noted  that  the  QA  work  force  had  received  little  management  attention  over 
the  years,  that  no  manning  baseline  existed  for  the  QA  effort,  and  that 
programs  were  experiencing  quality  problems.  He  tasked  Col  Micheal 
Nassr.  director  of  manufacturing  at  Headquarters  AFSC,  to  evaluate  the 
management  and  effectiveness  of  manning,  training,  methods,  regulations, 
contractual  requirements,  specifications,  and  overall  command  manage¬ 
ment  philosophy  of  the  QA  function. 

As  previously  mentioned,  Quality  '77  set  the  standard  for  many  studies 
to  follow.  A  team  of  about  10  AFSC  personnel  made  a  comprehensive  review 
of  20  reports  dating  back  to  1962.  A  few  of  the  common  findings  from  those 
reports  are  listed  below  and  can  be  used  to  measure  how  many  changes 
have  been  incorporated  into  the  quality  assurance  function: 

•  QA  not  considered  at  technical  reviews. 

•  Too  few  military  personnel  assigned  to  quality. 

•  A  manpower  baseline  not  set. 

•  Measurements  of  quality  not  specified. 

•  Contractual  incentives  for  reducing  failure  costs  not  utilized. 

•  Inspection  subcontractors  too  specific  and  too  often  duplicated. 

•  Contract  administration  organizations  (CAO)  too  often  base  QA  on 
emit  factor  performance. 
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The  study  team  also  compiled  and  reviewed  a  comprehensive  list  of  Depart  - 
ment  of  Defense  (DOD)  and  Air  Force  policies  and  regulations  on  quality 
assurance.  They  interviewed  more  than  100  DOD.  Air  Force,  and  industry 
personnel  at  various  levels,  including  the  Office  of  the  Secretary  of  Defense 
(OSD),  buying  organizations,  and  CAOs. 

The  Quality  '77  team  concluded  “that  quality  assurance  lacked  adequate 
leadership  and  suffered  from  a  poor  image  within  the  command.  Further¬ 
more,  personnel  manning  the  function  need  a  stronger  career  development 
program.”10  The  team  made  11  primary  and  six  secondary  recommenda¬ 
tions.  The  major  findings  were: 

1 .  Contractors  must  be  held  responsible  for  the  quality  of  their  products 
and  the  government  must  enforce  its  requirements  for  quality.  More  use 
of  contractual  rewards  and  penalties  was  needed  to  motivate  the  contractor 
to  produce  quality  products. 

2.  AFSC  had  no  clear  policy  statement  on  the  role  of  quality.  A  command 
policy  was  needed  that  identifies  contractual  requirements:  states  contrac¬ 
tor  and  government  responsibilities;  and  lists  of  quality  assurance  tasks. 

3.  The  QA  function  was  buried  as  a  subordinate  organization  in  the 
Directorate  of  Contract  Management  at  Headquarters  AFSC.  The  as¬ 
surance  disciplines  needed  to  be  integrated  into  one  functional  organization 
called  product  assurance,  reporting  directly  to  the  DCS  for  procurement 
and  manufacturing. 

4.  Neither  the  contractor  nor  the  government  was  using  quality  cost 
indicators  as  a  management  tool  even  though  contractually  required  on 
most  major  programs.  The  Air  Force  Contract  Management  Division 
(AFCMD)  and  the  system  program  offices  (SFO)  needed  to  develop  a  total 
quality  cost  management  program. 

5.  AFSC  had  no  disciplined  program  to  measure  a  contractor's  quality 
performance  to  collect  the  kinds  of  data  that  could  be  used  to  determine 
the  contractor’s  performance. 

6.  Because  of  the  lack  of  top  management’s  attention,  the  current  QA 
work  force  lacked  the  necessary  skills  to  accomplish  the  QA  mission. 
Training  and  career  development  for  both  military  and  civilian  workers  in 
the  quality  assurance  disciplines  required  immediate  attention. 

7.  QA  tasks  needed  to  be  identified  no  later  than  the  acquisition  phase 
by  the  system  program  office  and  contract  administration  organization  and 
should  be  used  by  manpower  to  develop  a  QA  manning  baseline. 1 1 


Quality  Horizons 

Gen  Alton  D.  Slay,  commander.  Air  Force  Systems  Command,  concerned 
with  the  significant  reduction  in  the  QA  work  force  and  impressive  advances 
in  aerospace  technology,  commissioned  the  Quality  Horizons  study  in 
November  1978.  He  tasked  Col  Bernard  L.  Weiss,  deputy  for  contracting 
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and  manufacturing.  Aeronautical  Systems  Division  (ASD),  to  determine  if 
AFSC  was  using  the  best  approach  to  ensure  quality  products.  The  study 
examined  the  following  four  areas: 

1.  Contractor  responsibility  for  end-item  quality  and  reducing  govern¬ 
ment  in-plant  presence. 

2.  Contractual  relationships  that  would  better  place  the  responsibility 
for  quality  with  the  contractor. 

3.  The  qualifications  and  any  changes  required  of  the  AFSC  quality 
assurance  work  force. 

4.  The  proper  QA  organization  and  structure  for  AFSC. 

The  study  team  consisted  of  about  10  people  representing  Headquarters 
AFSC,  AFCMD,  and  the  product  divisions.  The  team  visited  several  DOD 
acquisition  organizations  and  the  governments  and  firms  in  five  different 
countries.  A  few  important  observations  were: 

•  Product  quality  is  a  function  of  top  management’s  interest  in  quality. 

•  Successful  government  and  industrial  organizations  blend  the  as¬ 
surance  sciences. 

•  Quality  can  not  be  inspected  into  the  product,  yet  AFSC  places  more 
resources  and  emphasis  on  conformance  verification  than  on  design, 
process  control,  and  test  planning. 

•  Program  managers  do  not  want  reduced  in-plant  surveillance. 

A  few  of  the  primary  recommendations  made  to  the  AFSC  commander 
are: 


•  AFSC  needs  a  product  assurance  policy  requiring  government  and  top 
industry  management  personnel  to  ensure  that  proper  emphasis  is  given 
to  the  product  assurance  function.  Executive-level  training  programs  will 
be  required  to  show  the  benefits  and  risks  of  product  assurance  programs. 

•  The  product  assurance  functions  should  be  consolidated  into  one 
organization  at  the  headquarters  and  product  divisions. 

•  Formal  training  and  intern  programs  for  the  product  assurance  func¬ 
tion  need  to  be  institutionalized. 

•  More  use  of  contractual  techniques  to  motivate  the  contractor  to 
provide  better  quality  products  is  needed. 

•  AFSC  needs  to  implement  minimum  in-plant  surveillance  (MIPS) 
program  to  reduce  government  surveillance  with  those  contractors  who 
supply  high  quality  products.12 

In  January  1980  General  Slay  endorsed  five  key  recommendations. 

1.  Quality  needs  more  emphasis  from  top  management  in  AFSC. 

2.  Quality  must  be  stressed  up  front  in  the  design  phase. 

3.  AFSC’s  quality  assurance  work  force  must  be  strengthened. 

4.  Contract  incentives  could  be  used  to  enhance  quality. 
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5.  Quality  could  be  improved  by  proper  organizational  placement  within 
AFSC. 13 


Contract  Incentives  for  Product  Quality 

The  Air  Force  Business  Research  Management  Center  (AFBRMC),  Wright  - 
Patterson  AFB,  Ohio,  contracted  for  a  study  in  August  1979  to  examine 
motivational  aspects  of  the  US  Air  Force  contracting  techniques,  focusing 
specifically  upon  QA  and  reliability  policies  and  management  techniques 
and  practices.  International  Technology  Corporation,  Satellite  Beach. 
Florida,  performed  this  10-month  study  by  first  conducting  a  thorough 
literature  review  of  current  quality  and  reliability  policy  and  regulations. 
The  study  team  then  interviewed  many  high-level  DOD  and  industry 
acquisition  personnel  and  sent  out  a  questionnaire  to  obtain  an  in¬ 
dustrywide  viewpoint  on  quality  and  reliability  policy. 

International  Technology  concluded  that  three  major  QA  policy  and 
motivational  voids  existed  in  the  Air  Force.  First,  the  Air  Force  was  not 
maintaining  data  to  evaluate  a  contractor’s  quality  performance  in  develop¬ 
ing  and  producing  major  systems,  although  DODD  4155.1.  Quality  Pro¬ 
gram .  and  AFR  74-1,  Quality  Assurance  Program,  both  precluded  awarding 
contracts  to  firms  producing  unsatisfactory  quality.  An  objective  methodol¬ 
ogy.  reproduced  in  appendix  A,  was  developed  to  give  AFSC  personnel  a 
simple  set  of  criteria  by  which  to  evaluate  a  contractor’s  quality  perfor¬ 
mance.  Second,  the  Air  Force  had  not  developed  a  methodology  to  utilize 
profit  motivation  in  the  QA  program.  The  study  recommended  that  cost- 
plus,  award-fee  (CPAF)  contracts  be  used  to  motivate  contractors  to  improve 
quality  products.  A  sample  award-fee  plan,  reproduced  in  appendix  B,  was 
developed  for  AFSC  personnel  to  use  as  a  guide.  Third,  the  Defense 
Acquisition  Regulation  (DAR).  which  preceded  the  Federal  Acquisition 
Regulation  (FAR),  lacked  motivational  considerations  for  junior  and  nonsu- 
pervisory  personnel  who  could  directly  influence  product  quality  through 
attention  to  detail.  The  study  recommended  and  proposed  four  changes  to 
the  DAR  to  provide  improved  methods  to  motivate  contractor  employees.14 


Project  Quality 

Gen  Robert  T.  Marsh,  commander,  AFSC,  expressed  his  concern  that 
manufacturers  were  attempting  to  “inspect  in”  quality  alter  production 
rather  than  designing  it  into  the  product.  The  result  was  that  AFSC  has 
invested  heavily  in  source  inspection  systems  and  scrap,  rework,  and 
reinspection  facilities.  The  cost  of  maintaining  the  personnel  and  equip¬ 
ment  to  perform  the.se  tasks  in  what  General  Marsh  called  "hidden  factory” 
costs  could  no  longer  be  tolerated  by  the  Defense  Department. 
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In  September  1982  he  assigned  Anthony  J.  DeLuca.  principal  assistant 
in  the  DCS,  Contracting  and  Manufacturing  (HQ  AFSC/PP),  as  the  project 
director.  His  charier  was  to  determine  what  caused  these  hidden  factory 
costs  and  to  devise  contractual  procedures  to  motivate  contractors  to 
reduce  these  “wasteful  practices."  The  study  group’s  recommendations  are 
divided  into  six  categories  sequenced  across  the  contracting  cycle. 

Preproposal  Actions 

The  study  group  outlined  18  possible  courses  of  action  that  could  be 
implemented  in  the  preproposal  stage  of  the  contracting  process.  These 
steps  included  having  the  contractor  review  proposed  requirements  and 
identify  those  which  have  adverse  effect  on  quality;  ensuring  that  contrac¬ 
tors  can  accurately  track  and  analyze  costs  of  scrap,  repair,  retest,  reinspec¬ 
tion,  and  rework  (SR4)  data;  ensuring  that  statistical  process  controls,  vield 
rates,  and  manufacturing  variability  are  analyzed  and  controlled:  and 
increasing  the  use  of  award  fees  and  allowing  the  contractor  to  “fill  in"  the 
delivery  schedule  that  best  fits  the  plant  capability. 

Specification  and  Statement  of  Work  Preparation 

The  study  team  made  nine  recommendations  as  to  preparing  the 
specifications  and  statements  of  work.  They  included  requirements  for 
warranties,  statistical  process  control,  reviewing  quality  items  at  design 
reviews,  and  establishing  rigorous  reliability  demonstration  tests. 

Negotiation 

The  study  group  recommended  that  government  QA  personnel  be  in¬ 
cluded  in  contract  negotiations  and  that  quality  considerations  be  equal  to 
cost,  schedule,  and  performance  in  the  source  selection  process. 

Performance,  Administration  of  Contracts 

The  Project  Quality  team  made  eight  recommendations  with  respect  to 
contract  administration.  These  proposals  included  elevating  quality  to  the 
same  level  as  cost,  schedule,  and  performance  at  design  reviews  and  having 
quality  briefed  by  senior  contractor  quality  personnel.  In  addition,  material 
review  boards  should  address  why  a  nonconforming  part  was  produced, 
and  the  end-item  user  should  become  more  involved  by  periodically  visiting 
the  contractor’s  facilities. 

Policy 

The  study  group  made  nine  policy  recommendations.  It  stressed  the  need 
to  make  quality  a  major  consideration  for  contract  award  and  emphasizing 
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quality  early  In  the  program  development  process.  The  study  panel  argued 
that  contracts  should  require  companies  to  report  and  track  indicators  of 
quality.  Program  managers  should  be  graded  on  the  quality  of  their 
products  and  should  brief  the  status  at  program  and  commander  reviews. 

Other  Considerations 

The  group  recommended  that  quality  improvement  training  and  motiva¬ 
tional  program  costs  be  allowed  as  a  direct  contract  charge:  the  number 
and  complexity  of  specifications  be  reduced:  a  military  standard  on  statis¬ 
tical  process  control  be  written:  and  scrap,  rework,  and  repair  standards 
be  developed  on  a  commoditywide  basis. 1 


Accountable  Contract  Management,  AFCMD  Review 

Gen  Lawrence  A.  Skantz,  former  commander.  AFSC,  commissioned  a 
study  in  September  1984  to  evaluate  what  the  Air  Force  Contract  Manage¬ 
ment  Division’s  (AFCMD)  role  in  contract  management  should  be  and  how 
effectively  the  division  was  performing  this  mission.  Brig  (fen  Ken  Johnson, 
deputy  chief  of  staff,  acquisition  logistics  (HQ  AFSC),  was  appointed  as  the 
study  director.  He  was  assisted  by  a  very  senior  group  of  advisers  including 
five  retired  general  officers,  an  industry  vice  president,  and  an  assistant 
secretary  of  the  Air  Force. 

The  study  group  presented  seven  recommendations  to  General  Skantz. 
First,  the  study  team  concluded  that  an  adversarial  relationship  existed 
between  many  of  AFSC's  product  division  SPOs  and  AFCMD’s  plant  repre¬ 
sentative  offices.  At  issue  was  the  question:  Is  AFCMD’s  mission  an 
independent  check  and  balance  on  the  buying  division?  This  relationship 
resulted  in  a  fundamental  problem  of  accountability  affecting  quality, 
reliability,  maintainability,  and  producibility  of  the  products.  To  solve  this 
problem,  the  study  team  recommended  that  the  AFSC  commander  create 
an  accountability  chain  from  the  AFSC  commander  to  the  product  division’s 
program  directors  and  then  to  the  Air  Force  plant  representative  offices 
(AFPROs).  The  study  team  also  recommended  that  the  SPO  program 
directors  should  make  the  AFPROs  their  agents  in  the  plant. 

Second,  the  study  group  identified  a  need  to  upgrade  the  work  force.  The 
average  AFCMD  grade  was  9.47,  the  turnover  rate  was  high,  and  the  cross 
flow  of  assignments  between  AFPROs  and  SPOs  was  low.  The  study  team 
recommended  that  a  career  path  be  established  that  included  assignment 
cross  flow  and  that  AFCMD  establish  a  training  center  to  provide  the 
specialized  skills  needed. 

Third,  the  study  team  observed  that  despite  all  the  rhetoric  of  designing 
quality  into  the  product  the  Air  Force  was  still  tiying  to  inspect  it  in.  Quality 
assurance  was  not  as  important  as  schedule,  cost,  and  performance 


17 


considerations,  and  the  integration  of  QA  throughout  the  design  and 
manufacturing  process  needed  much  more  emphasis.  The  study  group 
recommended  that  the  Air  Force  stress  to  industry  that  quality  in  design  is 
mandatory.  Also.  AFCMD  needs  to  expand  the  use  of  contractor  operation 
reviews  using  more  SPO  personnel. 

Fourth,  the  study  team  noted  that  the  Air  Force  has  not  made 
producibility  in  design  a  high-priority  item,  in  fact  limited  government 
resources  were  devoted  to  this  function.  Most  SPO  engineering  resources 
are  performance  oriented.  The  group  also  observed  that  the  Air  Force  had 
no  way  of  obtaining  scrap,  rework,  and  repair  data  and  did  not  have  any 
way  of  getting  this  data  to  the  SPO.  The  study  team  recommended  that  the 
buying  divisions  needed  at  least  100  people  dedicated  to  ensuring  that 
quality  and  producibility  get  an  early  emphasis  in  the  design  phase.  Also, 
it  recommended  that  design  producibility  be  elevated  in  the  source  selection 
process  and  that  scrap,  rework,  and  repair  data  be  delivered  to  the 
government. 

Some  general  comments  made  by  the  study  team  were  that  the 
government’s  hardware  surveillance  role  was  only  marginally  effective  and 
incentives  should  be  used  to  motivate  the  contractor  to  do  the  job  right  the 
first  time  and  at  least  cost.  If  the  incentive  proved  effective,  the  government 
should  withdraw  from  the  hardware  surveillance  role. 16 


Efforts  to  Strengthen  DOD’s 
Quality  Assurance  Program 

In  November  1985  Sen  William  V.  Roth.  Jr.,  chairman.  Committee  on 
Government  Affairs,  asked  the  Government  Accounting  Office  (GAO)  to 
investigate  if  weaknesses  existed  in  the  various  DOD  contractor  surveil¬ 
lance  programs  and  determine  what  must  be  done  to  identify  and  initiate 
changes.  The  GAO  team  reviewed  many  quality  studies,  reports,  and 
articles  and  interviewed  many  DOD  and  contractor  personnel.  It  also 
visited  two  plant  representatives'  offices — one  Navy  and  one  Army — to 
review  their  plans  and  procedures  for  verifying  contractor  compliance  with 
contract  requirements. 

The  GAO  concluded  that  the  in-plant  quality  assurance  programs  were 
not  as  effective  as  they  should  be  in  ensuring  that  quality  products  are 
delivered  to  field  activities.  The  agency  found  that  the  plant  representative 
offices  failed  to  implement  fully  the  Defense  Department's  Federal  Acquisi¬ 
tion  Regulation  requirements  and  had  no  comprehensive  DOD-wide  plan 
for  improving  the  in-plant  QA  program.  The  GAO  investigators  showed  that 
neither  the  plant  representatives  nor  the  contractors  had  data  readily 
available  to  identify  recurring  contractor  deficiencies  even  though  extensive 
quality  data  existed.  Additionally,  government  personnel  were  not  perform¬ 
ing  all  required  government  inspections  to  verify  that  products  conformed 
to  contract  requirements.  Reviews  by  the  services  and  the  Defense  Logistics 
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Agency  (DLA)  contract  administration  organizations  identified  widespread 
contractor  quality  deficiencies. 

The  study  recommended  that  the  secretary  of  defense  direct  the  seivices 
and  DLA  to  implement  a  standardized  approach  for  contract  surveillance 
and  to  develop  a  long-range  plan  for  implement  mg  a  DOD-wide  plan  for 
assuring  optimum  quality.  This  recommendation  was  based  upon  DODD 
4155. 1 .  Quality  Assurance  Program,  which  requires  the  DOD  components 
to  develop  and  use  joint  procedures  for  QA  programs. 17 

Summary 

In  this  chapter,  the  author  reviews  six  major  studies  done  over  the  last 
10  years  that  focus  on  AFSC's  quality  assurance  function.  These  studies 
serve  as  a  foundation  for  this  current  study.  The  next  chapter  describes 
the  research  methodology  used  in  this  study. 
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CHAPTER  3 


Research  Methodology 

The  six  studies  discussed  in  chapter  2  provide  background  information 
on  the  status  of  quality  management  in  Air  Force  Systems  Command.  They 
also  serve  as  a  reference  point  for  determining  how  quality  management 
might  be  changing.  In  this  chapter  I  analyze  the  recommendations  of  these 
previous  studies  to  see  to  what  degree  those  recommendations  were  suc¬ 
cessfully  implemented  and  to  what  extent  deficiencies  still  exist. 


Description  of  Population 

The  literature  on  total  quality  management  suggests  that  everyone  in  the 
organization  can  influence  and  is  responsible  for  ensuring  a  quality 
product.  However,  in  this  study  I  limit  the  population  of  interest  to  three 
groups  of  acquisition  personnel.  The  first  group  includes  the  quality 
assurance  personnel,  both  managers  and  engineers,  whose  functional 
responsibility  is  quality  assurance.  The  second  group  of  interest  includes 
the  program  directors,  who  have  the  ultimate  responsibility  (referred  to  as 
program  managers  in  DOD  Directive  5000. 1 ,  Major  and  Non-Major  Defense 
Acquisition  Programs)  for  delivering  a  quality  product  to  the  user.  Group 
three  consists  of  the  program  or  project  managers,  who  work  directly  for 
the  program  directors  managing  many  of  the  subsystems  or  projects. 

Points  of  contact  at  each  of  the  five  AFSC  product  divisions  identified  185 
quality  managers  or  engineers  who  work  full-  or  part-time  in  the  quality 
discipline  on  one  or  more  programs  or  who  work  in  the  staff  function.  I 
surveyed  the  entire  population  of  quality  assurance  personnel. 

After  trying  unsuccessfully  to  get  a  list  of  program  directors  from  Head¬ 
quarters  AFSC,  I  reviewed  organizational  charts  from  each  product  division 
to  identify  the  program  directors  of  major  systems.  Using  the  five  product 
divisions’  organizational  charts  that  were  current  as  of  December  1987,  I 
identified  145  system  program  office  (SPO)  program  directors,  deputies,  or 
assistants.  Given  that  the  program  director's  decisions  have  a  major  effect 
on  quality,  I  surveyed  all  145  directors. 

Headquarters  AFSC,  Deputy  Chief  of  Staff,  Manpower  and  Personnel  (HQ 
AFSC/MP).  provided  a  complete  computer  list  of  program  or  project 
managers  in  the  rank  oflieutenant  colonel  through  second  lieutenant  with 
Air  Force  speciality  codes  271xand  272x.  The  list  contained  1,912  names; 
the  majority  were  located  at  the  five  product  divisions  of  interest  in  this 
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research.  After  reviewing  the  list.  I  reduced  this  subpopulation  to  1,402 
names  by  eliminating  those  personnel  located  at  other  bases  and  those  with 
nonprogram  or  nonproject-management  job  titles. 

The  total  population  for  this  research  was  approximately  1 ,732  subjects: 
1,402  program  or  project  managers,  145  program  directors,  and  185  QA 
personnel.  These  people  were  located  at  the  five  product  divisions  in  four 
different  states. 


Data  Collection 

Among  the  many  ways  to  survey  a  population  to  collect  data  are  personal 
interviews,  telephone  interviews,  and  mailed  questionnaires.  Although 
some  argue  that  personal  and  telephone  interviews  can  provide  more 
reliable  data,  when  samples  are  comparable  the  mode  of  data  collection 
does  not  affect  most  survey  estimates.  Personal  and  telephone  interviews 
have  the  disadvantage  of  high  cost,  including  travel  time  for  personal 
interviews  and  time  required  to  contact  each  individual.  The  most  sig¬ 
nificant  weakness  of  the  mail  survey  approach  is  the  nonresponse  bias. 
This  nonresponse  bias  can  change  the  survey  results  significantly  depend¬ 
ing  upon  the  percentage  of  those  not  responding  and  how  materially 
different  they  are  from  the  respondents  and  from  the  whole  population.  The 
size  of  the  population,  its  physical  dispersion  in  four  different  states,  and 
the  travel  constraints  imposed  by  this  research  program  dictated  the  use 
of  the  mail  survey  technique.  The  mail  survey  is  uniquely  qualified  to 
overcome  the  high  cost  and  time  required  for  personal  and  telephone 
interviews  and  also  has  the  advantage  of  giving  the  respondents  more  time 
to  collect  facts,  look  up  records,  and  give  more  thoughtful  answers.2 

I  surveyed  all  185  quality  assurance  personnel  and  145  program  direc¬ 
tors.  Because  of  the  large  number  of  program  or  project  managers  (1 ,402), 
I  chose  a  statistically  random  sample,  using  a  random  number  table  to 
select  640  names  from  the  list  provided  by  Headquarters  AFSC/MP.  This 
sample  represents  approximately  46  percent  of  the  AFSC  product  divisions’ 
program  or  project  managers.  This  sample  size  was  chosen  primarily  for 
economic  reasons  to  keep  the  total  number  of  surveys  mailed  below  1,000. 
Although  a  larger  sample  gives  a  better  estimate  of  the  population,  a  sample 
size  of  about  200  would  have  provided  a  95-percent  confidence  rate  with  a 
±6  percent  error  that  any  statement  made  about  the  population  from  the 
sample  mean  would  be  true.3  A  95-percent  confidence  rate  with  an  error 
rate  of  less  than  10  percent  is  often  used  to  provide  an  acceptable  level  of 
accuracy  for  this  data. 

Table  2  shows  the  number  of  surveys  mailed  to  each  product  division  by 
each  subpopulation  surveyed.  The  response  rate  from  each  subpopulation 
was  significantly  less  than  expected.  People  do  not  respond  to  mail  surveys 
for  many  reasons.  One  important  reason  is  poor  questionnaire  design, 
which  includes  poorly  written  questions  and  too  many  questions  that 
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require  a  great  deal  of  time  and  effort  to  complete.  Another  important 
reason  for  low  response  rate  deals  with  the  respondent’s  interest  in  the 
subject.  Unless  one  is  dealing  with  a  group  of  respondents  who  have 
genuine  interest  in  the  problem  under  investigation,  know  the  sender,  or 
have  some  common  bond  of  loyalty  to  a  sponsoring  institution  or  organiza¬ 
tion.  the  rate  of  returns  is  frequently  disappointing.4  People  respond  who 
have  a  particular  interest  in  the  subject.  The  actual  responses  for  each 
population  and  the  estimated  error  for  a  95-percent  confidence  rate  are 
listed  in  table  3.  Approximately  51  percent  of  the  quality  assurance 
personnel,  42  percent  of  the  program  directors,  and  34  percent  of  those 
program  or  project  managers  surveyed  responded  to  the  questionnaires. 

TABLE  2 

Number  of  Questionnaires  Mailed 


ESD 

ASD 

AD 

SD 

BMO 

Total 

Quality  Personnel 

60 

65 

13 

36 

11 

185 

Program  Directors 

37 

54 

12 

35 

7 

145 

Project  Manaaers 

97 

367 

57 

77 

42 

640 

TOTAL 

194 

486 

82 

148 

60 

970 

TABLE  3 

Responses  and  Estimated  Error  Rate 


Returns 

Error  Rate 

(n) 

(Rate) 

Quality  Personnel 

94 

(51%) 

7% 

Program  Directors 

61 

(42%) 

10% 

Project  Manaaers 

217 

(34%) 

6% 

TOTAL 

372 

Because  of  the  potential  problems  from  low  response  rates.  I  made  several 
attempts  to  assure  a  response  rate  over  50  percent.  1  tested  the  survey 
instrument  to  ensure  that  it  was  clear  and  concise  and  would  take  less  than 
30  minutes  to  complete.  Additionally,  to  emphasize  the  importance  of  the 
topic,  a  letter  (appendix  C)  from  the  AFSC  deputy  chief  of  staff  for  product 
assurance  (HQ  AFSC/PL)  introduced  the  questionnaire.  In  a  final  attempt 
to  increase  the  response  rate.  I  sent  a  follow-up  letter  (appendix  C)  to  the 
program  directors  and  quality  assurance  personnel.  This  letter  signifi- 
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cantly  increased  the  response  rate  of  the  program  directors  and  the  quality 
assurance  personnel.  A  follow-up  letter  was  not  sent  to  the  program 
managers  because  of  the  large  numbers  and  cost  involved. 


Development  and  Validation  of  Survey  Instrument 

As  stated  previously,  poor  questionnaire  design  can  result  in  unaccept¬ 
able  response  rates  that  can  invalidate  the  research  effort.  1  originally 
included  more  than  150  questions  in  this  survey  effort.  These  questions 
were  submitted  to  and  reviewed  by  personnel  at  AFSC’s  Aeronautical 
Systems  Division,  Directorate  of  Engineering  (ASD/EN),  Directorate  of 
Contracting  and  Manufacturing  (ASD/PMD),  and  Human  Resources 
Laboratory  (ASD/HRL).  I  also  sought  assistance  from  the  Air  Force  Institute 
ofTechnology  (AFIT)  and  Headquarters  AFSC,  Deputy  Chief  of  Staff,  Product 
Assurance  (HQ  AFSC/PLEQ).  Next  1  prepared  a  draft  questionnaire  of  66 
questions,  which  incorporated  the  inputs  from  these  reviews.  I  then  sent 
a  revised  questionnaire  to  these  same  organizations  for  review.  After 
incorporating  several  added  changes,  I  provided  a  copy  to  the  Air  University. 
Deputy  Chief  of  Staff.  Operations  and  Plans  (AU/XPZ),  for  review.  Person¬ 
nel  in  this  office  assisted  in  developing  the  five-point.  Likert-type  scale  used 
for  the  majority  of  the  questions.  Finally.  I  developed  another  draff  ques¬ 
tionnaire  with  74  questions,  incorporating  AU/XPZ’s  recommendations  for 
the  program  directors  and  managers. 

1  selected  four  students  attending  Air  Command  and  Staff  College, 
Maxwell  AFB.  Alabama,  who  had  previous  program  management  ex¬ 
perience,  to  pretest  the  questionnaire.  They  were  asked  not  only  to  com¬ 
plete  the  questionnaire  but  to  evaluate  the  questionnaire’s  structure: 

•  How  long  did  it  take  to  answer  the  questionnaire? 

•  Are  the  questions  clear  and  without  assumptions? 

•  Are  there  any  terms  that  could  be  misinterpreted? 

•  Are  any  questions  biased? 

•  Are  the  instructions  clear? 

All  four  ACSC  students  returned  the  questionnaires  and  provided  com¬ 
ments  that  were  incorporated  into  the  final  draft. 

I  modified  the  first  section  of  the  program  director  and  manager  ques¬ 
tionnaire  to  get  additional  information  about  the  background  and  educa¬ 
tional  experience  of  the  quality  assurance  personnel.  This  additional 
background  information  is  the  primary  difference  between  the  two  ques¬ 
tionnaires  developed  for  this  research  project.  Both  questionnaires  were 
sent  to  the  Air  Force  Manpower  and  Personnel  Center.  Personnel  Survey 
Branch  (HQ  AFMPC/DPMYOS)  through  AU/XPZ  for  approval  to  survey  Air 
Force  civilian  and  military  personnel.  They  requested  only  editorial  and 
typographical  changes.  Formal  approval  was  received  from  Headquarters 
AFMPC/DPMYOS  by  a  letter  dated  2  December  1987. 
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Description  of  Survey  Instrument 

The  program  director  and  project  manager’s  questionnaire  is  included  as 
appendix  D  and  the  questionnaire  for  quality  assurance  personnel  as 
appendix  E.  As  stated  in  the  previous  section  the  primary  difference 
between  the  two  questionnaires  is  in  the  work  force  profile,  which  provides 
general  background  information.  Question  25  on  the  quality  personnel 
questionnaire  corresponds  to  question  12  on  the  program  director  ques¬ 
tionnaire.  The  reason  for  using  identical  questions  was  to  compare  the 
responses  from  each  subpopulation. 

Both  questionnaires  contain  six  sections  or  groups  of  questions.  The  first 
section  requests  background  information  on  rank,  experience,  education, 
and  training.  This  information  is  used  to  categorize  the  three  survey  groups 
and  compare  how  the  quality  assurance  work  force  has  changed.  The  next 
section  (questions  13-21  on  the  program  director  survey)  deals  with  the 
cost-of-quality  concept  and  asks  respondents  to  identify  costs  for  each 
component  of  quality  (i.e.,  failures,  inspection,  and  prevention).  The  pur¬ 
pose  of  those  questions  is  to  determine  if  people  understand  and  receive 
cost-of-quality  data  and  if  they  use  it  to  manage  their  programs.  The  third 
group  of  questions  (12,  22-50)  identifies  those  items  that  can  improve  or 
impair  quality.  This  group  of  questions  seeks  to  determine  how  prevalent 
those  practices  are  in  Air  Force  quality  assurance  programs.  The  fourth 
section  (questions  51-60)  identifies  indicators  of  quality  that  should  be 
analyzed  by  government  and  contractor  personnel  to  control  the  cost  of 
quality.  The  fifth  group  of  questions  (61-70)  identifies  specific  approaches 
that  government  and  contractor  personnel  can  use  to  achieve  better  quality. 
The  last  section  (questions  153-158)  asks  respondents  to  rank  the  top  three 
contributors  to  and  detractors  from  achieving  high  quality  and  productivity. 

Selection  of  Statistical  Tests 

The  Statistical  Package  for  Social  Sciences-X  (SPSS-X)  was  used  for  the 
analysis  of  the  survey  data.  Since  most  of  the  questions  used  a  nominal 
Likert-type  scale,  nonparametric  statistical  tests  were  used  to  analyze  the 
data.  However,  some  parametric  tests,  such  as  the  t  test  and  analysis  of 
variance  (ANOVA)  test,  were  used  because  of  the  additional  information 
provided  by  these  tests,  but  only  in  conjunction  with  appropriate  non 
parametric  tests 

One  of  the  simpler  techniques  used  to  analyze  the  survey  data  was  to 
review  the  distribution  of  responses  for  each  question.  Especially  important 
were  those  items  where  a  significant  number  of  respondents  were  grouped 
toward  the  extreme  ends  of  the  scale.  For  example,  question  50  hi  the 
program  director  survey  identified  that  more  than  60  percent  of  those  who 
responded  felt  that  Headquarters  AFSC’s  leadership  was  contributing  very 
little  or  not  at  all  to  improving  the  quality  of  their  system.  The  analysis 
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technique  of  combining  the  two  outside  categories  of  a  live-point  Likert  scale 
is  described  by  John  W.  Best  and  James  V.  Kahn.5 

The  mode,  another  useful  statistical  measurement,  was  used  to  identify 
which  response  was  most  frequently  chosen  for  each  question.  The  mean 
or  arithmetic  average  was  not  used  since  much  of  the  data  was  coded  on  a 
nominal  scale.  The  frequency  distributions  of  each  subpopulation  were  also 
compared  for  agreement  among  and  between  subpopulations.  Several 
different  statistical  tests  were  used  to  determine  what  appeared  to  be  a  lack 
of  agreement  between  subpopulations.  These  tests  are  described  in  the 
following  paragraphs. 

The  cross  tabulations  (CROSSTABS)  available  in  SPSS-X  was  also  used 
to  test  how  the  responses  from  one  question  might  alTect  another  question. 
For  example,  would  military  or  civilian  rank  affect  how  one  answered  a 
question?  Question  2  in  the  quality  assurance  personnel  survey  asked  for 
the  respondent's  military  rank.  This  information  was  used  (o  compare  how 
military  rank  affected  the  response  to  various  questions  (e.g..  question  43, 
the  benefits  of  an  award-fee  contract  to  improve  product  quality).  This  test 
was  veiy  helpful  in  identifying  how  certain  subsets  within  subpopulations 
respond  to  questions.  An  important  question  to  ask  is  whether  the  three 
subpopulations  agreed  on  various  questions.  Another  interesting  question 
is  whether  on  average  military  and  civilian  QA  personnel  respond  the  same 
way.  Many  statistical  tests  are  available  in  SPSS-X  to  perform  these  types 
of  analysis. 

The  KruskaTWallis  one-way  analysis  of  variance  by  ranks  is  one  of  the 
most  useful  and  reliable  nonparametric  tests  available  to  determine  if  there 
is  agreement  among  three  or  more  different  subpopulations.  With  more 
than  360  total  responses  in  this  research  effort,  the  likelihood  of  total 
agreement  is  small.  But  the  question  is  whether  there  really  is  a  significant 
difference  among  the  subpopulations  or  whether  the  differences  are  just 
those  expected  from  a  random  sample.  Compared  with  the  one-way 
analysis  of  variance  (ANOVA)  Ftest,  the  Kruskal-Wallis  test  has  a  power 
efficiency  of  about  96  percent.6  Although  the  data  does  not  satisfy  the 
assumptions  of  parametric  statistics,  the  Ftest  was  run  to  see  if  the  results 
would  be  the  same.  Since  the  Ftest  is  a  more  powerful  test,  running  the  F 
test  would  give  additional  confidence  if  both  tests  provided  the  same  results. 
This  test  determined  how  well  the  program  directors,  program  managers, 
and  quality  assurance  personnel  agreed  on  various  subjects.  One  would 
think  that  the  philosophy  of  the  program  director  would  permeate  the  entire 
organization  and  there  would  tend  to  be  agreement  among  these  subpopula¬ 
tions. 

The  Mann-Whitney  U  test  was  used  to  test  whether  two  independent 
groups  were  from  the  same  population.  It  was  used  specifically  to  see  if 
military  and  civilian  quality  assurance  personnel  were  in  agreement  with 
how  they  answer  survey  questions.  (Again,  one  would  hope  that  quality 
assurance  personnel  as  a  whole  would  be  using  similar  management 
philosophies  throughout  AFSC.)  This  test  is  one  of  the  most  powerful 
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nonparametric  tests  and  the  best  alternative  to  the  l  test,  a  parametric 
statistic  for  comparing  samples  from  a  population.  When  compared  with 
the  powerful  t  test,  the  Mann-Whitney  test  has  a  power  efficiency  of  about 
95  percent.7  As  before,  the  t  test  was  run  to  compare  the  results  with  the 
Mann-Whitney  U  test  in  order  to  obtain  additional  confidence  with  this  test . 

Questions  153-155  provided  a  list  of  10  items  and  asked  the  respondents 
to  rank  the  three  most  important  contributors  in  improving  quality.  A 
similar  request  was  made  in  questions  156-158,  which  asked  the  respon¬ 
dents  to  rank  the  three  most  important  items  that  detract  from  achieving 
quality.  The  data  from  these  questions  was  run  on  the  SPSS-X  software 
program  called  Multi-Response.  This  program  rank  orders  the  responses 
by  frequencies  and  percentage  of  total  responses.  This  output  provides  a 
ranking  of  what  people  think  are  the  most  important  contributors  and 
detractors  of  quality  products.  The  results  of  these  te^ts  are  presented  in 
chapter  4  and  the  conclusions  and  recommendations  dra  ”u  rrom  the  data 
are  presented  in  chapter  5. 


Notes 

1.  Floyd  J.  Fowler.  Jr..  Survey  Research  Methods  (Beverly  Hills.  Calif.:  SAGE  Publica- 
tions,  1984).  66. 

2.  Ibid  .  71. 

3.  Based  on  a  standard  error  of  proportion  where: 

N  =  population  size  =  1000 
n  =  sample  size  =  200 
7  =  confidence  =  ±1 .96  for  95  percent 
d  =  error  rate 

p  =  estimate  of  actual  population  (set  to  0.5  as  a  conservative  estimate) 

q  =  (1-p) 
and 

pq  N  -  n 

d  =  z  n  N  -  1 

thus  for  this  example,  d  =  0.062  or  error  rate  ±6  percent. 

For  the  QA  respondents  (where  N  =  185  and  n  =  94).  d  =  0.071  or  error  rate  of  +7  percent. 
For  the  program  directors  (where  N  =  145  and  n  =  61),  d  =  0.096  or  error  rate  of  ±10 
percent. 

For  the  program  managers  (where  N  =  1402  and  n  =  2 17).  d  =  0.061  or  error  rate  of  ±6 
percent. 

4.  John  W.  Best  and  James  V.  Kahn,  Research  in  Education  (Englewood  Cliffs,  N.J.: 
Prentice- Hall.  1986).  166. 

5.  Ibid..  181. 

6.  Sidney  Siegel,  Nonparametric  Statistics  for  the  Behavioral  Sciences  (New  York:  Mc- 
Graw  Hill.  1956).  192-93. 

7.  Ibid.,  126. 
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CHAPTER  4 


Findings 

In  this  chapter  I  summarize  the  results  of  investigating  the  six  research 
questions  presented  in  chapter  one.  First.  I  give  a  demographic  analysis  of 
the  acquisition  work  force  showing  the  characteristics  of  each  subpopula¬ 
tion:  program  directors  (PD),  program  managers  (PM),  and  quality  as¬ 
surance  (QA)  workers.  Then  I  note  important  recommendations  from 
previous  studies  that  relate  to  this  research  study.  In  the  remainder  of  the 
chapter.  I  show  the  responses  of  each  subpopulation  to  a  specific  research 
question.  I  provide  the  raw  data  in  the  following  appendixes:  program 
directors  in  appendix  F,  program  managers  in  appendix  G,  QA  military 
personnel  in  appendix  H.  and  QA  civilian  personnel  in  appendix  I. 

Demographic  Data:  Program  Directors,  Program 
Managers,  and  Quality  Assurance  Personnel 

The  rate  of  response  to  the  survey  questionnaires  was  good  enough  to 
permit  valid  inferences  from  the  results.  Of  the  145  program  directors 
queried  at  the  five  AFSC  product  divisions.  61  (42%)  returned  fully  com¬ 
pleted  surveys.  Out  of  the  sample  of  640  program  managers  at  these  same 
product  divisions,  217  (34%)  responded.  I  queried  185  QA  personnel  at 
these  product  divisions:  94  (51%)  returned  the  questionnaires. 

Table  4  shows  the  frequency  distribution  by  the  respondent's  rank  or 
grade.  Seventy-one  percent  (71%)  of  the  program  directors  are  colonels  or 
GM-15s.  Sixty  percent  (60%)  of  the  program  managers  are  captains  or 
equivalent  or  less,  which  indicates  a  young,  probably  inexperienced,  pro¬ 
gram  management  work  force.  Eighty  percent  (80%)  of  the  civilian  QA 
personnel  are  in  the  grade  of  GM-12  or  -13  and  74%  of  the  military  QA 
personnel  are  lieutenants. 

TABLE  4 

Rank  and  Grade  Frequency  Distribution 


Category 

PD 

Absolute 

Frequency 

(n) 

PM 

QA 

PD 

Relative 

Frequency 

(%) 

PM 

QA 

Brig  Gen,  GM-16,  or  higher 

7 

3 

0 

11.5 

1.4 

0.0 

Col  or  GM-15 

43 

3 

0 

70.5 

1.4 

0.0 

Lt  Col  or  GM-14 

10 

40 

5 

16.4 

18.4 

5.3 

Maj  or  GM-13 

0 

41 

26 

0.0 

18.9 

27.7 

Capt,  GM-12,  or  lower 

1 

130 

63 

1.6 

59.9 

67.0 

TOTAL 

6? 

217 

94 

100.0 

100.0  100.0 

~  —  —  —  — 

25  =  =  = 

=  =  = 

—  —  = 

=  =  =r  = 

=  = 
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Table  5  Indicates  the  years  of  experience  in  the  program  management 
function.  More  than  70%  of  the  program  directors  have  10  or  more  years 
of  experience  in  program  management.  A  cross-tabulation  analysis  on  the 
program  director’s  data  indicates  that  100%  of  the  brigadier  generals,  81% 
of  the  colonels,  and  67%  of  the  lieutenant  colonels  or  equivalents  have  eight 
or  more  years  of  program  management  experience.  Only  12%  of  the 
program  managers  have  10  or  more  years  of  experience,  while  73%  have 
five  years  or  less  of  experience.  A  cross  tabulation  shows  that  all  the 
brigadier  generals,  33%  of  the  colonels.  5 1%  of  the  lieutenant  colonels,  20% 
of  the  majors,  and  7%  of  the  captains  have  eight  or  more  years  of  experience 
in  program  management. 

TABLE  5 

Number  of  Years  in  Program  Management 


Category 

Absolute 

Frequency 

(n) 

PD  PM 

Relative 

Frequency 

(%) 

PD  PM 

10  years  or  more 

43 

26 

7 1.7 

12.0 

8-9  years 

5 

17 

8.3 

7.8 

6-7  years 

3 

17 

5.0 

7.8 

4-5  yeais 

3 

42 

5.0 

19.4 

3  or  less 

6 

115 

10.0 

53.0 

TOTAL 

60 

217 

100.0 

100.0 

Table  6  shows  the  number  of  years  that  QA  personnel  have  worked  in 
quality.  Twenty-five  percent  (25%)  of  the  civilians  have  worked  1 8  or  more 
years  while  19%  have  worked  in  quality  no  more  than  three  years.  Ninety- 
two  percent  (92%)  of  the  military  QA  personnel  have  no  more  than  three 
years  of  quality  experience.  This  finding  also  indicates  a  very  young  and 
inexperienced  work  force. 

TABLE  6 

Number  of  Years  in  Quality  Assurance 


Category 

Absolute 

Frequency 

(n) 

MIL  CIV 

Relative 

Frequency 

(%) 

MIL  CIV 

1 8  years  or  more 

0 

17 

0.0 

25.4 

1 3-1 7  years 

0 

8 

0.0 

11.9 

8-12  years 

0 

13 

0.0 

19.4 

4—7  years 

2 

16 

8.0 

23.9 

3  or  less 

23 

13 

92.0 

19.4 

TOTAL 

25 

67 

100.0 

100.0 

=  =  =  =  = 

=  =  ==  = 

Table  7  indicates  the  number  of  different  systems  or  items  (hat  the 
respondents  have  worked  on.  Almost  60%  of  the  program  directors.  25% 


of  the  program  managers,  and  65%  of  the  QA  personnel  have  worked  on 
five  or  more  systems.  However,  46%  of  the  program  managers  and  45%  of 
military  QA  personnel  are  working  on  their  first  or  second  system.  The 
cross  tabulation  shows  that  among  program  directors  71%  of  the  brigadier 
generals.  65%  of  the  colonels,  and  30%  of  the  lieutenant  colonels  or 
equivalents  have  managed  five  or  more  systems.  For  program  managers. 
100%  of  the  brigadier  generals,  67%  of  the  colonels,  35%  of  the  lieutenant 
colonels.  15%  of  the  majors,  and  23%  of  the  captains  or  equivalent  have 
managed  five  or  more  systems. 

TABLE  7 

Number  of  Systems  or  Items  Managed 


Category 

PD 

Absolute 

Frequency 

(n) 

PM 

QA 

PD 

Relative 

Frequency 

(%) 

PM  QA 

5  or  more  systems/items 

36 

54 

60 

60.0 

25.1 

64.5 

4  systems/items 

8 

22 

5 

13.3 

10.2 

5.4 

3  systems/items 

8 

40 

11 

13.3 

18.6 

11.8 

2  systems/items 

4 

50 

7 

6.7 

23.3 

7.5 

1  system/item 

4 

49 

10 

6.7 

22.8 

10.8 

TOTAL 

60 

215 

93 

100.0 

100.0 

100.0 

II 

II 

II 

II 

II 

II 

II 

II 

II 

=  =  =  = 

====  = 

=  =  = 

—  —  — 

=  = 

Table  8  indicates  the  number  of  years  that  the  respondents  have  worked 
on  their  current  program.  It  shows  that  more  than  70%  of  the  program 
directors  and  managers  and  56%  of  the  QA  personnel  have  less  than  two 
years  experience  on  the  current  program. 

Table  9  indicates  the  acquisition  phases  of  the  system  or  item  for  which 
the  respondents  are  primarily  responsible.  Ninety-three  percent  (93%)  of 
the  program  directors,  84%  of  the  program  managers,  and  84%  of  the  QA 
personnel  are  working  on  programs  in  the  full-scale  development,  produc¬ 
tion,  or  deployment  phases.  The  cross  tabulation  for  program  directors 
indicates  that  only  two  colonels  and  two  lieutenant  colonels  are  working  on 
programs  in  the  conceptual  and  demonstration-validation  phases.  A  cross 
tabulation  for  program  managers  shows  that  no  colonels  and  only  20%  of 
the  lieutenant  colonels,  15%  of  the  majors,  and  14%  of  the  captains  or 
equivalents  are  responsible  for  programs  in  the  conceptual  and  demonstra¬ 
tion-validation  phases.  Some  respondents  indicate  that  they  are  respon¬ 
sible  for  systems  in  several  phases  and  therefore  have  difficulty  answering 
specific  questions.  In  this  situation,  most  provide  what  they  think  is  an 
average  response  for  all  their  systems.  Some  respondents  indicate  that 
their  programs  are  in  the  conceptual  or  demonstration-validation  phases 
and  they  feel  many  questions  are  not  applicable  and  therefore  either  left 
the  question  blank  or  did  not  complete  the  questionnaire.  (These  blank 
answers  are  not  used  in  the  analysis.) 
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Based  upon  comments  received,  I  suspect  that  many  of  the  question¬ 
naires  were  not  returned  because  of  an  apparent  common  misconception 
that  the  quality  function  has  little  role  to  play  in  the  early  acquisition 
phases.  However,  the  questionnaire  was  designed  around  the  total  quality 
management  (TQM)  philosophy,  which  should  not  be  constrained  by  ac¬ 
quisition  phases.  This  lack  of  response  could  indicate  that  a  significant 
number  of  respondents  do  not  understand  the  TQM  approach. 

TABLE  8 

Number  of  Years  on  Current  Program 


Category 

PD 

Absolute 

Frequency 

(n) 

PM 

QA 

PD 

Relative 

Frequency 

(%) 

PM 

QA 

5  or  more  years 

5 

5 

11 

8.2 

2.3 

12.1 

4  years 

5 

13 

18 

8.2 

6.1 

19.8 

3  years 

8 

47 

11 

13.1 

22.0 

12.1 

2  years 

13 

63 

35 

21.3 

29.4 

38.5 

1  year 

30 

86 

16 

49.2 

40.2 

17.6 

TOTAL 

6? 

214 

9? 

100.0 

100.0 

100.1 

TABLE  9 
Acquisition  Phase 


Category 

PD 

Absolute 

Frequency 

(n) 

PM 

QA 

PD 

Relative 

Frequency 

(%) 

PM 

QA 

Conceptual 

2 

6 

3 

3.3 

2.8 

3.4 

Demonstration-Validation 

2 

27 

11 

3.3 

12.7 

12.5 

Full-scale  Production 

28 

100 

40 

45.9 

47.2 

45.5 

Production 

25 

48 

30 

41.0 

22.6 

34.1 

Deployment 

4 

31 

4 

6.6 

14.6 

4.5 

TOTAL 

61 

212 

88 

100.0 

99.9  100.0 

Tabie  10  indicates  how  many  full-time  quality  assurance  personnel  are 
assigned  to  the  organization.  Approximately  33%  of  the  program  directors, 
31%  of  the  program  managers,  and  39%  of  the  QA  personnel  report  that 
they  have  four  or  more  full-time  QA  personnel  assigned.  A  significant 
number  of  the  program  directors  (16%)  and  program  managers  (26%)  say 
they  have  no  full-time  QA  assigned.  These  individuals  are  totally  respon¬ 
sible  for  all  aspects  of  quality  on  these  programs,  yet  they  have  had  very 
little  education  or  training  in  quality  (table  1 1). 
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TABLE  10 

Full-time  Quality  Assurance  Personnel  Assigned 


Category 

PD 

Absolute 

Frequency 

(n) 

PM 

QA 

PD 

Relative 

Frequency 

(%) 

PM 

QA 

4  or  more  assigned 

20 

64 

35 

32.7 

30.7 

38.9 

3  assigned 

3 

16 

17 

4.8 

7.7 

18.9 

2  assigned 

10 

26 

13 

16.8 

12.4 

14.4 

1  assigned 

18 

49 

16 

29.4 

23.4 

17.8 

0  assigned 

10 

54 

9 

16.3 

25.8 

10.0 

TOTAL 

6? 

209 

90 

100.0 

100.0  100.0 

Almost  half  (43%)  of  the  program  directors  and  more  than  half  (64%)  of 
the  program  managers  have  had  little  training  and  education  in  quality 
(table  1 1).  The  cross  tabulation  for  program  directors  indicates  that  50% 
of  the  brigadier  generals  and  47%  of  the  colonels  or  equivalent  have  had 
nine  hours  or  less  of  training  in  quality  improvement.  For  the  program 
managers  100%  of  the  brigadier  generals.  67%  of  the  colonels.  43%  of  the 
lieutenant  colonels.  55%  of  the  majors,  and  73%  of  the  captains  or 
equivalents  have  had  nine  hours  or  less  of  such  training.  Yet.  23%  of  the 
program  directors  and  32%  of  the  program  managers  spend  more  than  two 
hours  a  day  (20%  or  more  of  their  time)  on  quality  issues  such  as  failures 
or  corrections  (table  12). 


TABLE  1 1 

Training  or  Education  on  How  to  Improve  Quality 


Category 

Absolute 

Frequency 

<n) 

PD  PM 

Relative 

Frequency 

(%) 

PD  PM 

40  or  more  hours 

9 

23 

15.0 

10.7 

39-30  hours 

3 

7 

5.0 

3.3 

29-20  hours 

11 

18 

18.3 

8.4 

1 9-1 0  hours 

11 

29 

18.3 

13.5 

9  or  less  hours 

26 

138 

43.3 

64.2 

TOTAL 

60 

215 

999 

100.1 

Table  1 2  also  reveals  that  a  very  large  percentage  of  the  program  directors 
(48%)  and  program  managers  (34%)  are  spending  at  least  an  hour  every  day 
making  decisions  affecting  quality.  Since  they  probably  have  not  had  the 
education  or  training  to  help  make  these  decisions,  the  question  to  be  asked 
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is:  Do  they  have  the  right  information  to  make  that  decision?  Answering 
this  question  is  part  of  the  objective  of  this  research. 

TABLE  12 

Hours  Spent  on  Quality  Issues 


Category 

Absolute 

Frequency 

(n) 

PD  PM 

Relative 

Frequency 

<%) 

PD  PM 

5  or  more  hours 

0 

10 

0.0 

4.8 

4-3  hours 

4 

23 

6.7 

11.0 

2  hours 

10 

33 

16.7 

15.7 

1  hour 

29 

71 

48.3 

33.8 

0  hour 

17 

73 

28.3 

34.7 

TOTAL 

60 

210 

100.0 

100.0 

When  asked  about  the  type  and  value  of  quality  education  and  training. 
66%  of  the  civilians  and  44%  of  the  military  of  the  QA  personnel  said 
adequate  opportunities  are  available  to  attend  formal  courses.  Neverthe¬ 
less.  46%  of  the  civilians  and  85%  of  the  military  have  had  less  than  five 
weeks  of  training.  Only  37%  of  the  civilians  and  10%  of  the  military  say 
the  training  has  helped  a  great  or  very  great  extent  on  I  heir  job.  The  survey 
asks  to  what  extent  certain  topics  have  been  covered  in  formal  courses  and 
how  important  they  might  be  on  the  job.  These  topics  include  manufactur¬ 
ing  processes,  cost-of-quality  concepts,  testing  procedures,  contract  re¬ 
quirements.  and  statistical  process  control  (SPC).  Thirty-seven  percent 
(37%)  of  the  civilians  and  70%  of  the  military  say  they  have  had  no  SPC 
courses  in  the  last  five  years.  Approximately  55%  of  military  and  civilians 
indicate  that  the  listed  topics  are  covered  very  little  or  not  at  all  in  the 
courses.  Seventy  percent  (70%)  of  the  civilians  and  55%  of  the  military  also 
indicate  that  these  topics  would  be  very  valuable  in  their  job. 


Summary  of  Demographic  Data 

Based  on  the  responses,  the  “typical"  program  director  is  a  colonel  or 
GM-15  with  more  than  10  years  of  program  management  experience  who 
has  managed  over  five  systems  and  has  been  on  the  current  program  less 
than  three  years.  He  or  she  has  had  a  lot  of  program  management  training 
(more  than  10  weeks)  but  little  on  quality  or  product  performance  and 
durability.  His  or  her  system  is  in  the  full-scale  development  or  production 
phase  and  he  or  she  has  one  to  two  quality  personnel  assigned  full  time. 
Only  six  percent  (6%)  of  the  program  directors  were  in  the  conceptual  or 
demonstration -validation  phases,  yet  16%  report  no  QA  personnel  assigned 
full  time.  This  means  at  least  10%  of  AFSC’s  programs  have  no  QA 
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personnel  assigned  full  time  in  the  full-scale  development  (FSD)  or  produc¬ 
tion  phase. 

The  "typical”  program  manager  was  a  captain  or  GM-12  with  less  than 
three  years  of  program  management  experience.  He  or  she  has  worked  on 
one  or  two  systems  and  has  been  assigned  to  the  current  program  less  than 
three  years.  He  or  she  has  had  six  to  seven  weeks  of  program  management 
training  but  little  on  quality  or  product  performance  and  darability.  His  or 
her  system  is  in  the  full-scale  development  or  production  phase  and  has 
one  or  two  full-time  QA  personnel  assigned.  As  with  the  program  directors, 
only  a  few  (16%)  of  the  program  managers  were  in  the  conceptual  or 
demonstration-validation  phases,  and  26%  reported  that  no  QA  personnel 
are  assigned  full  time  to  these  phases  of  the  acquisition  process. 

A  “typical"  QA  civilian  is  an  engineer  in  grade  of  GM-12  or  -13  with  10 
years  of  experience.  He  or  she  has  worked  on  five  systems  and  been  on  the 
current  program,  which  is  in  the  FSD  phase,  about  three  years.  He  or  she 
works  with  two  to  three  other  QA  personnel.  Training  time  is  available 
about  two-thirds  of  the  time  and  he  or  she  may  have  taken  one  SPC  course 
and  one  product  performance  course  in  the  last  five  years.  However,  about 
40%  have  had  neither  of  these  courses. 

A  “typical"  military  QA  person  is  an  engineer  in  the  grade  of  first 
lieutenant  with  three  or  fewer  years  of  experience.  He  or  she  has  worked 
on  three  systems  and  has  been  on  the  current  program,  which  is  also  in 
the  FSD  phase,  about  two  years.  He  or  she  either  works  alone  or  has  three 
to  four  QA  personnel  assigned  to  the  program.  The  respondents  indicate 
that  opportunities  for  training  are  available  only  about  half  as  often  as  they 
should  be.  He  or  she  probably  has  not  received  any  SPC  courses,  only  one 
product  performance  course,  and  no  more  than  Five  weeks  of  QA  training. 

Given  the  response  rate  to  the  questionnaires,  I  can  say  with  a  relatively 
high  level  of  confidence  that  the  results  accurately  reflect  the  thinking  of 
people  in  the  AFSC  acquisition  community  as  to  how  well  they  are  control¬ 
ling  and  reducing  the  hidden  factory  costs.  The  next  step  is  analyzing  each 
of  the  six  main  lines  of  inquiry  in  greater  depth. 

How  Have  Past  Studies  Addressed 
Total  Quality  Management? 

One  purpose  oft  his  research  is  to  see  to  what  extent  past  Air  Force  studies 
have  addressed  ideas  of  total  quality  management  (TQM).  A  second  objec¬ 
tive  is  to  determine  from  the  responses  to  the  survey  questions  whether  the 
findings  of  these  previous  studies  are  still  relevant  today.  Five  of  the  six 
previous  studies  reviewed  identify  the  need  to  use  more  positive  incentives 
to  motivate  contractors  to  improve  quality. 

The  respondents  in  the  present  research  are  asked  to  rank  positive 
incentives  as  to  their  importance  to  improving  quality.  The  respondents 
select  this  item  as  the  sixth  most  important  contributor  to  improving 
quality.  Seven  survey  questions  ask  which  method  works  best  to  influence 
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the  contractor  to  provide  better  quality-  These  options  included  using 
award  fees,  warranties,  firm-fixed -price  contracts,  lop  management  involve¬ 
ment,  contract  requirements,  employee  motivation  techniques,  and  good 
business  practices.  Most  respondents  indicate  that  firm-fixed -price  con¬ 
tracts  do  very  little  to  motivate  contractors  to  provide  better  quality. 
Warranties  and  award  fees  are  the  two  methods  the  respondents  select  as 
having  the  greatest  influence  on  the  contractor.  The  program  directors  are 
consistently  more  optimistic  than  the  program  managers  and  QA  personnel 
on  the  effectiveness  of  these  incentives  as  motivators.  One  program  director 
says  that  “DOD  .  .  .  under  emphasized  [the  potential  of  using]  positive 
incentives  to  [get]  the  job  [done]  right  the  first  time." 

The  need  for  more  QA  personnel  and  for  more  formal  training  programs 
is  also  identified  in  three  of  these  studies.  Approximately  50%  of  the 
respondents  in  the  current  research  study  indicate  that  the  system  program 
offices  (SPO)  do  not  have  enough  QA  personnel  assigned.  The  following  are 
a  few  examples  of  many  written  comments  provided  by  the  respondents. 

A  program  director  said.  “Large  shortage  of  experienced  quality  people." 

A  program  manager  said,  "require  SUFFICIENT.  EXPERIENCED.  COMMUTED. 
PEOPLE." 

A  prognun  director  said.  “I'm  hiring  a  non  personal  service  contractor,  to  support  SIX) 
QA  people." 

Moreover,  almost  half  of  the  QA  work  force  say  that  adequate  oppor¬ 
tunities  to  attend  formal  courses  were  not  available.  They  also  indicate  that 
the  value  of  the  courses  they  are  able  to  attend  is  questionable.  The 
respondents  state  that  these  courses  included  little  material  on  statistical 
process  control,  cost-of-quality  concepts,  and  other  topics  that  would  be 
very  valuable  in  performing  their  jobs.  The  following  are  a  few  written 
comments  provided  by  the  respondents. 

A  program  manager  said.  "AFSC  training  is  a  shambles  not  meeting  the  needs  of  new 
managers  .  .  .  ignoring  the  need  to  train  quality  managers  up  front  just  like 
maintenance  officers." 

A  QA  manager  said.  “QA  personnel  for  the  Government  should  be  forced  to  attend  QA 

classes." 

A  program  manager  said,  "need  more  QA  classes  ai  AFIT.  .  .  .  We  need  quality  people 
and  education.” 

A  third  recommendation  that  all  six  of  the  past  studies  make  is  the  need 
to  obtain  eost-of-qualily  data  and  use  it  to  measure  contractor  performance. 
Research  questions  five  and  six  specifically  address  to  what  degree  the  SPO 
and  contractor  are  using  this  information.  The  findings  of  the  current  study 
indicate  that  approximately  half  of  the  work  force  are  still  not  reviewing  or 
using  this  data  even  though  it  is  required  by  many  government  regulations. 
As  shown  by  the  following  comments,  the  respondents  fail  to  understand 
the  value  of  this  data,  how  to  use  it.  and  even  what  data  to  obtain. 

A  program  manager  said,  “No  hardware  yef  to  measure  its  quality." 
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A  program  director  said.  "Just  started  (fabrication)  of  test  article."  therefore,  no  scrap, 
rework,  repair,  or  cost  of  quality  data. 

A  business  manager  (27xx)  said,  "responsible  for  analysis  of  contractor  total  program 
cost  and  schedule  variance  data  .  .  .  not  close  enough  to  quality  speciality  to  really 
have  much  insight  on  how  it  operates." 

A  program  manager  said.  "Quality  needs  to  be  measurable  throughout  the  weapon 
system  acquisition  process." 

A  program  director  said.  “I  do  not  know  how  to  quantify  these  costs.  .  .  .  Spacecraft 
program  .  .  .  long  life  without  repair." 

A  program  manager  on  a  software  program  said,  "hardware  items,  scrap,  etc.,  are  not 
applicable." 

As  many  of  these  comments  indicate,  several  respondents  think  of  quality 
as  an  inspection  function  to  be  done  by  quality  assurance  personnel  on 
hardware  in  the  production  phase.  According  to  TQM,  this  perception  must 
be  changed  if  the  hidden  factory  costs  are  going  to  be  reduced. 

Four  of  the  studies  see  the  need  for  top  management  to  commit  itself  to 
accepting  quality  products.  Most  of  the  respondents  rate  the  commitment 
to  not  accepting  poor  quality  as  the  second  most  important  contributor  to 
quality.  Other  survey  data  from  the  current  study  indicates  that  quality  is 
a  very  important  objective  in  the  SPO  but  is  not  receiving  the  resources 
commensurate  with  this  ranking.  The  respondents  also  say  that  the  staff 
QA  personnel  and  Headquarters  AFSC  leadership  are  doing  very  little  to 
improve  quality.  The  following  comments  seem  typical  of  what  the  work 
force  thinks  of  top  management's  commitment  to  quality. 

A  program  manager  said,  "senior  leadership  displays  a  lack  of  integrity  in  excuses 
given  for  failure  and  poor  planning/ decisions  on  their  part." 

A  program  manager  said.  “Problem  is  the  lack  of  ability  to  make  a  decision  by  upper 
level  personnel  .  .  .  they  are  afraid  that  it  (higher  quality)  can  t  be  done." 

A  program  manager  said.  “If  the  MAJCOM  and  Air  Staff  were  really  serious,  they  would 
provide  the  required  resources." 

A  QA  manager  said.  “DOD  and  AFSC  does  little  to  provide  the  resources  and 
organizational  structure  for  QA." 


Two  of  the  past  studies  stress  that  the  importance  of  quality  assurance 
must  be  increased  to  be  equal  with  cost,  schedule,  and  performance. 
Respondents  in  this  study  rank,  the  low  priority  the  Air  Force  gives  to 
program  objectives  as  the  fourth  most  important  detractor  to  quality  out  of 
the  10  listed  in  the  questionnaire.  Most  program  directors  and  managers 
indicate  that  schedules  have  affected  quality  to  a  very  little  extent .  However, 
only  about  20%  of  the  QA  personnel  agree.  A  QA  manager  with  30  years  of 
experience  comments  that  “major  programs  (are]  giving  lip-service  to 
quality."  Additionally,  the  data  indicates  that  the  value  or  effectiveness  of 
the  QA  personnel  is  not  highly  rated.  This  finding  may  result  because  of 
too  little  manpower,  lack  of  training,  or  some  other  factors.  Hov  ever, 
visibility  does  not  appear  to  be  a  problem  to  the  program  director  but  could 
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be  with  program  managers.  A  program  manager  comments  that  “quality 
personnel  seem  to  be  very  unaggressive  or  .  .  .  unaware  of  their  role." 

What  Are  the  Most  Important 
Factors  Affecting  Quality? 

A  second  purpose  of  this  research  is  to  determine  what  the  acquisition 
work  force  thinks  are  the  most  important  contributors  in  improving  quality 
and  productivity  by  asking  the  respondents  to  rank  the  top  three  factors 
from  a  list  of  10  items  that  affect  quality.  These  items  are  then  compared 
to  the  responses  of  over  20  survey  questions  to  assess  what  the  acquisition 
personnel  are  actually  doing.  The  three  subpopulations  show  strong 
agreement  in  how  they  rank  many  of  the  factors  that  contribute  to  better 
quality  (table  13).  The  top  three  are  early  inputs  into  the  design  process 
(producibility,  manufacturing,  and  quality):  commitment  to  not  accepting 
poor  quality;  and  program  stability  (including  funding,  requirements,  and 
the  design). 

TABLE  13 

Contributors  to  Better  Quality 


QA  Personnel 


Program  Directors 


Program  Managers 


No  Poor  Quality 
Early  Design  incut 
Control  Requirement 
Top-Down  Training 
Program  Stability 
Disallow  SRR  Cost 
Use  Past  Perform 
Positive  Reinforce 

Using  Templates - 

Refer  Discrepancies- 


Early  Design  Input 
No  Poo>  C'LShty  — 
Program  Stability  - 
Top-Down  Training 
Dcsitive  Reinforce 
Control  Requirements' 
Use  Past  Perform 
Disallow  SRR  Cost 
Using  Templates 
Refer  Discrepancies' 


Early  Design  Input 
No  Poor  Quality 
Program  Stability 
Control  Requirements 
Top-Down  Training 
Use  Past  Perform 
Positive  Reinforce 
Disallow  SRR  Cost 
Refer  Discrepancies 
Using  Templates 


Two  questions  directly  relate  to  what  the  respondents  chose  as  the  most 
important  contributor  to  quality.  The  first  question  asks  respondents  to 
what  extent  they  are  using  a  development  process  that  designs  quality  into 
the  product.  Approximately  60%  of  the  program  directors,  30%  of  the 
program  managers,  and  30%  of  the  QA  personnel  say  they  are  using  this 
process  a  great  to  very  great  (Great-V.G.)  extent  (fig.  1).  However.  25%  of 
the  QA  and  35%  of  the  program  managers  either  do  not  answer  (because 
t  hey  do  not  know)  or  say  that  they  were  not  using  this  process  at  all  or  were 
using  it  very  little  (Not  Answ-V.L.).  The  important  indicators  or  trends  with 
respect  to  a  TQM  philosophy  should  lie  at  the  extremes  of  the  scale.  For 
example,  if  30%  answer  they  are  using  this  process  a  lot  (Great-V.G.)  and 
30%  say  to  some  degree  (Some),  then  one  could  surmise  that  60%  are  using 
this  process  at  least  to  some  degree  and  could  conclude  that  this  response 


38 


rate  is  good.  However,  does  60%  using  this  process  some  represent  a  true 
commitment  to  TQM?  One  could  reasonably  argue  that  to  achieve  a  high 
level  of  quality  and  commitment  to  TQM  70-80%  of  the  responses  must  be 
in  the  great  to  very  great  category. 

Figure  2  shows  to  what  extent  respondents  believe  producibilitv  risk 
reduction  efforts  are  well  funded  before  production.  Funding  of  such 
quality  assurance  items  is  important  to  ensuring  that  quality  is  incor¬ 
porated  early  in  the  design  process.  Forty  to  fifiv  percent  (40-50%)  of 
respondents  indicate  that  producibility  efforts  are  not  adequately  funded 
to  reduce  risk  and  assure  quality.  Designing  quality  into  the  product  is  no 
doubt  one  of  the  most  important  elements  in  achieving  a  quality  product. 
The  respondents  recognize  that  up-front  funding  of  producibility  efforts  can 
increase  quality,  but  they  also  indicate  this  funding  is  not  happening. 

Another  important  contributor  to  improving  quality  and  productivity  is  a 
commitment  to  not  accepting  poor  quality.  Four  of  the  survey  questions 
measure  this  attitude.  One  question  asks  respondents  whether  producing 
a  quality  product  for  the  user  is  the  organization's  most  important  organiza¬ 
tional  objective.  Program  directors  feel  that  quality  is  a  very  important 
objective,  but  only  about  50%  of  the  program  managers  and  40%  of  the  QA 
personnel  respond  positively  (fig.  3). 

A  second  indicator  of  the  importance  of  quality  is  how  much  of  its 
resources  an  organization  is  expending  to  improve  quality.  About  40%  of 
the  program  managers  and  QA  personnel  respond  that  their  agency  spends 
too  few  of  its  resources  on  improving  quality  (fig.  4).  Approximately  40%  of 
the  program  directors  say  that  sufficient  resources  are  being  expended, 
which  does  not  correlate  with  their  statement  that  quality  is  the  most 
important  objective.  If  that  is  the  most  important  objective,  resources  can 
and  will  be  found. 

Two  survey  questions  provide  an  indication  of  how  well  the  work  force  is 
monitoring  data  that  identifies  poor  quality.  Figure  5  shows  the  results  of 
asking  to  what  extent  contractors  use  material  review  board,  quality 
deficiency  report  (MRB/QDR)  actions  as  a  measure  or  indicator  of  quality. 
Only  30%  of  the  program  managers  and  QA  personnel  respond  that  the 
contractor  is  using  this  information  to  a  significant  extent.  An  even  more 
alarming  statistic  is  that  20%  of  the  program  directors  and  QA  personnel 
and  35%  of  the  program  managers  do  not  answer  this  question  (because 
they  do  not  know).  In  response  to  a  question  about  the  extent  to  which  they 
monitor  defects  and  workmanship  data,  about  50%  of  the  program  directors 
and  QA  personnel  say  that  they  were  doing  this  a  lot  but  only  30%  of  the 
program  managers  agreed  (fig.  6).  By  ranking  this  item  as  the  second  most 
important  contributor  to  improving  quality,  the  work  force  shows  that  it 
knows  the  importance  of  having  a  commitment  to  quality.  However,  the 
data  shows  that  this  commitment  is  more  rhetoric  than  practice. 

The  respondents  rank  top-down  corporate  training  to  cultivate  awareness 
of  QA's  impact  on  life-cycle  costs  as  the  fifth  most  important  contributor 
improving  quality.  Two  questions  measure  this  belief.  One  question  asks 
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Quality— MOST  Important  Objective.  Figure  4.  Expending  Sufficient  Resources. 


to  what  extent  SPOs  have  educational  programs  aimed  at  excellence  in 
management  and  technical  fields.  Almost  50%  of  the  respondent  indicate 
that  the  SPO  is  doing  very  little  in  this  area  (fig.  7).  A  second  question  asks 
to  what  extent  contractor's  top  management  puts  emphasis  on  the  quality 
of  the  product.  Only  30%  of  the  program  managers  and  QA  personnel 
indicate  that  management  is  doing  a  lot  in  this  area  (fig.  8).  The  program 
director’s  response  also  identifies  the  significant  disagreement  between 
them  and  the  program  managers  and  QA  personnel.  The  ranking  of  this 
item  indicates  that  respondents  recognize  the  importance  of  top  manage¬ 
ment  training  and  involvement  in  achieving  a  quality  product.  However, 
the  SPO’s  top  management  does  not  seem  to  be  committed  to  training,  and 
the  contractor’s  top  management  does  not  appear  to  make  a  significant 
commitment  to  influencing  quality. 

The  respondents  rank  positive  reinforcement  programs  (incentives)  as  the 
sixth  most  important  contributor  in  improving  quality.  One  question  asks 
to  what  extent  would  an  award-fee  contract  influence  a  contractor  to  provide 
a  quality  product.  Only  35%  of  the  program  managers  and  QA  personnel 
believe  award-fee  contracts  would  do  a  lot  to  get  better  quality:  vet  more 
than  50%  of  the  program  directors  believe  that  this  incentive  would  work 
with  their  contractor  (fig.  9).  A  second  question  asks  to  what  extent  would 
a  firm- fixed -price  (FFP)  contract  influence  contractors  to  provide  a  quality 
product.  More  than  50%  of  the  respondents  said  that  an  FFP  contract 
would  do  very  little  to  achieve  better  quality  (fig.  10). 

fhe  program  directors  are  significantly  more  positive  than  program 
managers  and  QA  personnel  in  believing  that  positive  reinforcement  or 
incentives  are  effective  motivators.  However,  most  respondents  indicate 
that  an  FFP  contract  does  no*  motivate  a  contractor  to  provide  bet  ter  quality. 

The  respondents  rank  the  disallowing  of  scrap,  rework,  and  repair  costs 
low  as  an  incentive  for  better  quality.  Two  survey  questions  reveal  why  they 
put  this  item  so  low.  When  asked  how  often  they  personally  review  a 
contractor's  rework,  repair,  retest,  and  scrap  levels,  more  than  50%  of  the 
program  managers  and  20%  of  the  QA  personnel  and  about  20%  of  the 
program  directors  say  they  never  review  this  data  (fig.  11).  The  next 
question  asks  how  often  do  they  personally  review  a  contractor's  “cost  of 
quality"  (failure  costs  +  routine  inspection  +  prevention  costs)  data  and 
trends.  More  than  40%  of  the  program  directors  and  QA  personnel  and 
60%  of  the  program  managers  never  review  this  data  (fig.  12). 

As  a  second  step  in  determining  what  factors  affect  quality,  the  respon¬ 
dents  rank  the  top  three  items  from  a  list  of  10  that  detract  from  achieving 
quality  and  productivity  (table  14).  The  top  three  are  unrealistic  program 
schedules,  lack  of  cooperation  among  personnel  (design,  quality,  test, 
manufacturing,  etc  ),  and  program  instability. 

A  related  question  in  the  survey  asks  the  respondents  to  what  extent  have 
delivery  schedules  taken  priority  over  quality  decisions.  Approximately 
50%  of  the  program  directors  and  managers  sav  that  schedules  have  had 
little  effect  on  quality,  but  only  20%  of  the  QA  personnel  agree  (fig.  13).  The 
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Figure  7.  Education  Programs  Aimed  at  Excellence  in  Figure  8.  Top  Management  Influencing  the 

Management  and  Technical  Fields,  Quality  of  Your  Products. 


program  managers  ranked  this  item  as  the  number  one  detractor  of  quality 
and  program  directors  ranked  it  number  four,  while  40%  of  the  QA 
personnel  indicated  that  schedules  have  greatly  affected  quality  (table  14). 


TABLE  14 

Detractors  from  Better  Quality 


QA  Personnel 

Program  Schedules 
Priority  of  Quality 
Lack  of  Cooperation 
Poor  Vendors/Subs 
Old  Quality  Approach 
Nebulous  Definition 
Complexity  of  DOD 
Inadequate  Investment 
Program  Stability 
Reassign  of  People 


Program  Directors 

Lack  of  Cooperation 
Inadequate  Investment 
Program  Stability 
Program  Schedules 
Poor  Vendors/Subs 
Old  Quality  Approach 
Nebulous  Definition 
Priority  of  Quality' 
Complexity  of  DOD 
Reassign  of  People 


Program  Managers 

Program  Schedules 
Program  Stability 
Lack  of  Cooperation 
Nebulous  Definition 
Priority  of  Quality 
Complexity  of  DOD 
Inadequate  Investment 
Reassign  of  People 
Poor  Vendors/Subs 
Old  Quality  Approach 


Lack  of  cooperation  among  personnel  is  the  second  greatest  detractor  of 
quality,  and  two  survey  questions  ask  what  the  respondents  are  actually 
doing  to  reduce  this  problem.  Approximately  60%  of  the  program  directors, 
but  only  30%  of  program  managers  and  QA  personnel,  say  they  devote  a 
lot  of  time  to  attempting  to  blend  the  SPO.  contract  administration  service 
(CAS),  and  contractor  into  a  team  (fig.  14).  This  question  again  indicates  a 
significant  difference  in  opinion  among  the  program  directors,  the  program 
managers,  and  QA  personnel.  A  second  question  asks  respondents  to  what 
extent  contractors  use  design  teams  to  improve  quality.  Approximately  45% 
of  the  program  managers  and  QA  personnel  indicate  that  they  are  ac¬ 
complishing  very  little  in  this  area  (fig.  15).  Twenty-five  percent  (25%)  of 
the  program  managers  and  10%  of  the  program  directors  and  QA  personnel 
do  not  answer  this  question  (or  do  not  know  the  answer).  The  responses 
to  these  two  questions  indicate  that  perhaps  the  communication  flow  in  the 
program  office  is  not  what  it  should  be.  The  functional  areas,  the  SPO.  the 
CAS,  and  the  contractor  historically  experience  some  level  of  conflict. 1 

The  priority  of  quality  within  Air  Force  program  objectives  ranks  fourth 
as  a  detractor.  Some  acquisition  personnel  say  quality  is  very  important, 
but  most  do  not  think  their  organization  is  devoting  enough  of  its  resources 
to  improving  quality.  Almost  50%  of  the  respondents  indicate  that  too  few 
people  are  assigned  to  the  QA  function  (fig.  16).  As  past  studies  reveal,  the 
priority  of  quality  has  been  a  concern  for  many  years  and  is  still  a  concern 
among  QA  personnel. 

Program  managers  and  quality  assurance  workers  rank  inadequate 
investment  as  the  sixth  most  important  item,  but  the  program  directors 
chose  this  item  as  the  second  greatest  detractor  from  quality.  The 
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responses  to  the  survey  Indicate  that  quality  is  not  receiving  the  up-front 
support  required  to  address  this  issue  (figs.  2  and  16). 

Program  managers  rank  poor  vendor  and  subcontractor  quality  levels  as 
the  seventh  most  significant  detractor,  while  QA  personnel  and  program 
directors  rank  this  item  fourth  and  filth  respectively.  Fifty-five  percent 
(55%)  of  the  program  managers  and  45%  of  the  QA  personnel  responded 
that  the  contractor  was  doing  very  little  to  develop  active  subcontractor 
quality  improvement  programs.  About  15%  of  program  directors  and  QA 
personnel  and  30%  of  the  program  managers  do  not  answer  this  question 
or  do  not  know  (fig.  17).  Regarding  the  extent  to  which  the  respondents  use 
subcontractor  yield  rates.  50%  of  the  program  directors  and  QA  personnel 
and  65%  of  the  program  managers  indicate  very  little,  if  at  all  (fig.  18).  A 
significant  number — 25%  of  the  QA  personnel,  15%  of  the  program  direc¬ 
tors,  and  20%  of  the  program  managers — say  they  do  not  use  subcontractor 
yield  data  at  all.  Only  30%  of  the  program  directors  and  QA  personnel  say 
that  the  contractor  is  using  a  preferred  vendor  program  with  ratings  on 
quality  and  schedule  to  a  great  extent  (fig.  19).  Again,  10%  of  the  QA 
personnel,  20%  of  the  program  directors,  and  30%  of  the  program  managers 
fail  to  answer.  A  final  question  in  this  area  asks  to  what  extent  does  your 
contractor  emphasize  the  supplier’s  quality.  The  responses  show  little 
agreement  among  the  program  director,  program  manager,  and  QA  person¬ 
nel  (fig.  20).  Approximately  50%  of  the  program  directors,  25%  of  the 
program  managers,  and  30%  of  the  QA  personnel  say  that  the  contractor 
emphasizes  supplier’s  quality  a  lot.  Again.  15%  of  the  program  directors 
and  QA  personnel  and  25%  of  the  program  managers  fail  to  answer  this 
question. 

One  research  question  asks  individuals  in  the  acquisition  work  force  to 
rank  a  list  of  10  items  that  can  improve  quality  and  a  similar  list  of  items 
that  can  detract  from  quality.  These  items  are  then  compared  with  other 
survey  questions  to  determine  if  they  are  accomplishing  those  tasks  which 
affect  quality  and  productivity  the  most.  Most  in  the  population  indicate 
that  they  are  not  utilizing  concepts  that  can  increase  quality  and  produc¬ 
tivity  to  a  great  extent.  The  data  also  indicates  that  many  respondents 
cannot  or  do  not  answer  many  questions  relating  to  how  quality  and 
productivity  can  be  improved  and  that  the  three  subpopulations  disagree 
significantly  on  many  questions.  This  finding  indicates  a  lack  of  com¬ 
munication  within  the  SPO. 


What  Is  the  SPO  Doing  to  Affect  Quality? 

A  third  purpose  of  this  research  is  to  measure  at  what  level  the  acquisition 
work  force  is  performing  certain  tasks  that  increase  the  level  of  quality.  This 
research  analyzes  four  aspects  of  this  question:  (1)  effectiveness  of  the  QA 
organization  in  the  SPO.  (2)  effectiveness  of  motivational  techniques  to 
improve  quality,  (3)  impact  of  quality  problems  on  program  management. 
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Figure  19.  Contractor  Using  a  Preferred  Vendor  Program  Figure  20.  Contractor  Emphasis  on  Supplier’s  Quality 

for  Quality  and  Schedule. 


and  (4)  utilization  of  the  quality  improvement  process.  The  survey  ques¬ 
tionnaire  includes  30  questions  relating  to  this  research  objective. 

Effectiveness  of  Quality  Assurance  in  the  SPO 

Between  40%  and  50%  of  the  respondents  say  that  QA  personnel 
contribute  only  to  some  extent  in  achieving  a  quality  product  (fig.  2 1).  This 
outcome  is  not  a  strong  endorsement  of  the  QA  personnel,  though  the 
program  directors  were  significantly  more  positive  than  the  program 
managers.  The  program  directors  indicate  an  extremely  positive  com¬ 
munication  channel  between  the  program  directors  and  QA  personnel,  but 
only  50%  of  the  program  managers  give  this  response  (fig.  22).  This  result 
indicates  that  although  channels  are  open  communication  may  not  be 
taking  place.  The  responses  to  other  survey  questions  indicate  that  each 
subpopulation  has  a  different  knowledge  level  or  perspective  concerning 
many  quality  issues. 

About  30—40%  of  the  respondents  are  very  positive  about  the  kind  of  QA 
personnel  in  their  organization  (fig.  23).  However.  30%  of  the  QA  personnel 
and  40%  of  the  program  managers  indicate  they  do  not  have  the  right  kind 
of  QA  personnel  working  on  their  programs.  About  20%  of  the  program 
managers  do  not  answer  this  question.  More  than  40%  of  the  respondents 
think  that  they  do  not  have  the  right  number  of  QA  personnel  assigned  (fig. 
16).  This  is  understandable  when  16%  of  the  program  directors  and  26% 
of  the  program  managers  say  they  have  no  full-time  QA  personnel  assigned. 

Forty  to  sixty  percent  (40-60%)  of  the  respondents  say  that  the  QA  staff 
provides  little  help  in  getting  a  better  product  (fig.  24).  Approximately  75% 
of  the  respondents  think  that  Headquarters  AFSC  is  doing  little  to  help 
achieve  a  quality  product  (fig.  25).  Many  respondents  feel  the  QA  organiza¬ 
tion  is  only  somewhat  effective  in  helping  to  achieve  a  quality  product — 
probably  because  of  the  lack  of  manpower,  but  lack  of  skills  may  also  be  a 
contributor.  The  data  also  strongly  indicates  that  Headquarters  AFSC  and 
the  staff  QA  organizations  need  to  reassess  what  they  are  doing  for  the 
SPOs. 

Contract  Incentives  and  Improved  Quality 

Approximately  65%  of  the  program  directors  and  45%  of  the  program 
managers  and  QA  personnel  say  that  contract  requirements  are  an  excellent 
way  to  influence  the  contractor  to  improve  quality.  However.  50%  of  the 
QA  personnel  also  say  that  the  contractor  can  be  influenced  only  to  some 
extent  using  contract  requirements  as  a  motivating  technique  (fig.  26). 
Fifty  five  percent  (55%)  of  the  program  directors,  40%  of  the  program 
managers,  and  25%  of  the  QA  personnel  indicate  (hat  the  contractor  can 
be  influenced  a  great  deal  by  good  business  practices  (fig.  27) — again,  not 
a  lot  of  agreement  among  the  respondents. 

Approximately  30%  of  the  respondents  indicate  that  the  user  has  very 
little  influence  on  the  quality  of  hardware  and  services  acquired  (fig.  28). 
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Figure  23  Right  Kind  of  QA  Personnel  Figure  24.  Staff  QA  Organization  Helps  Achieve 

Assigned  to  the  SPO.  a  Quality  Product. 


100 


S' 


50 


Figure  27.  Influence  the  Contractor  Using  Figure  28.  User's  Influence  on  Contractor’s 

Good  Business  Practices.  Quality  of  Hardware/Services. 


An  equal  number  of  the  program  directors  and  managers  indicate  that  the 
user  influences  quality  a  great  deal.  However,  only  1 5%  of  the  QA  personnel 
give  this  response. 

When  asked  whether  an  award-fee  or  FFP  contract  would  influence 
contractors  to  provide  a  quality  product,  more  than  50%  of  program 
directors  indicate  an  award  fee  would  greatly  influence  the  contractor  (figs. 
9  and  10).  The  respondents  are  very  consistent  in  indicating  that  an  FFP 
contract  would  do  very  little  to  influence  the  contractor  to  provide  a  quality 
product.  Approximately  40%  of  the  respondents  say  that  warranties  would 
greatly  influence  their  contractor  (flg.  29).  However,  30%  of  the  program 
managers  and  QA  personnel  indicate  that  a  warranty  would  do  very  little 
to  influence  the  contractor. 

About  50%  of  the  program  directors  and  managers  feel  that  good  contract 
requirements  and  business  practices  can  motivate  the  contractor  to  provide 
a  quality  product.  The  QA  personnel  agree  that  good  contract  requirements 
can  motivate  the  contractor  but  they  do  not  agree  that  good  business 
practices  are  as  effective.  Many  responses  indicate  that  the  user  is  not  very 
effective  in  influencing  quality  and  possibly  should  become  more  involved. 
Approximately  40%  of  the  respondents  indicate  that  warranties  and  award 
fees  would  greatly  influence  the  contractor  and  more  than  50%  say  that 
FFP  contracts  would  do  very  little  to  influence  the  contractor  to  provide  a 
quality  product. 

Impact  of  Quality  Problems 

The  respondents  are  asked  to  what  extent  the  number  of  quality  problems 
in  the  last  year  are  normal  for  their  program.  Approximately  40%  of  the 
respondents  say  they  are  having  the  normal  amount  of  problems  for 
programs  as  complex  as  theirs  (fig.  30).  In  a  related  survey  question,  30% 
say  they  have  had  no  significant  quality  problems  which  impacted  cost  or 
schedule,  or  caused  an  accident,  or  gave  unwanted  publicity  (fig.  48). 
However,  40%  say  they  have  had  from  one  to  three  significant  problems. 

Fifty  percent  (50%)  of  the  respondents  indicate  t  hat  manufacturing  errors 
have  some  impact  on  managing  the  system  while  45%  of  the  program 
managers  indicate  these  errors  have  very  little  effect  (fig.  31).  Many 
respondents  say  that  engineering  errors  cause  some  problems,  but  30% 
reported  that  they  have  very  little  impact  on  managing  the  program  (fig.  32). 

The  responses  to  questions  about  the  extent  to  which  design  and 
manufacturing  problems  impact  schedule,  cost,  performance,  and 
reliability  (fig.  33)  show  two  things.  First,  the  QA  personnel  say  in  sig¬ 
nificantly  higher  numbers  than  program  directors  or  managers  that  these 
items  are  greatly  affected  by  design  and  manufacturing  problems.  Second, 
approximately  50%  of  the  QA  respondents  indicate  that  design  and 
manufacturing  problems  have  greatly  affected  schedule,  but  the  program 
directors  and  managers  indicate  that  schedule  has  not  affected  quality  of 
the  product  (fig.  13).  In  short,  program  directors  and  managers  are  saying 
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Figure  33.  Design/Manufacturing  Problems  Effect  on  Schedule,  Cost,  Performance,  or  Reliability. 


quality  decisions  are  not  being  overridden  by  schedule;  QA  personnel  do 
not  agree,  however.  With  respect  to  the  extent  to  which  the  SPOs  use 
hardware  quality  audits,  about  45%  of  the  respondents  indicate  that  they 
are  using  these  audits  a  lot  and  another  30%  indicate  they  use  audits  at 
least  some  of  the  time  (fig.  34).  Although  most  in  the  QA  work  force  are 
aware  of  the  quality  of  their  products,  the  data  does  not  indicate  any 
excellence  in  the  product. 

The  majority  of  respondents  indicate  that  they  are  not  having  major 
quality  problems  but  do  have  some  manufacturing  and  engineering 
problems.  The  majority  also  indicate  that  design  and  manufacturing 
problems  have  little  impact  on  schedule,  cost,  performance,  and  reliability. 
However,  schedule  and,  to  a  lesser  degree,  cost  are  affected  to  a  greater 
extent  than  performance  and  reliability.  This  belief  may  indicate  only  that 
cost  and  schedule  are  more  visible  and  easier  to  measure  than  performance 
and  reliability. 

Using  Quality  Improvement 

The  respondents  are  also  asked  to  assess  to  what  degree  they  are  using 
a  continuous  quality  improvement  (QI)  process.  Quality  improvement  is  as 
much  a  philosophy  as  it  is  a  practice,  and  it  must  start  with  a  serious 
commitment  from  top  management.  More  than  70%  of  the  program  direc¬ 
tors  indicate  that  quality  is  the  most  i’  iportant  program  objective  (fig.  3). 
However,  only  50%  of  the  program  managers  and  40%  of  the  QA  personnel 
agree.  If  the  program  directors  are  committed  to  high  quality,  they  have  to 
communicate  that  commitment  to  the  workers.  One  way  is  through 
educational  programs.  However,  50%  of  the  program  managers  and  QA 
personnel  indicate  that  the  SPOs  use  few  educational  programs  aimed  at 
excellence  in  management  and  technical  fields  (fig.  7).  Also  the  SPOs  must 
communicate  this  commitment  to  the  contractors.  The  program  directors 
are  significantly  more  positive  than  the  program  managers  and  QA  person¬ 
nel  in  saying  that  they  stressed  quality  objectives  in  dealing  with  contractors 
(fig.  35).  More  than  75%  of  the  program  directors  indicate  that  the 
contractor  does  share  the  same  quality  objectives  but  less  than  50%  of  the 
program  managers  and  QA  personnel  agree.  Most  program  directors  (65%) 
thought  a  lot  of  team  building  was  taking  place  in  the  system  program  office 
(SPO),  in  contract  administration  service  (CAS),  and  at  the  contractor,  but 
less  than  40%  of  the  program  managers  and  QA  personnel  agree. 

If  SPOs  are  committed  to  improving  quality,  they  must  set  measurable 
goals  against  which  to  judge  progress.  Yet.  more  than  50%  of  the  respon¬ 
dents  indicate  that  the  SPOs  have  veiy  few  measurable  goals  (fig.  36).  When 
asked  to  what  extent  the  SPO  is  monitoring  performance  to  goals,  ap¬ 
proximately  45%  of  the  respondents  say  that  it  is  doing  so  to  a  great  extent 
(fig.  37).  This  result  is  interesting  because  the  respondents  previously  say 
that  they  have  few  measurable  goals.  Approximately  30%  of  the  respon¬ 
dents  say  they  are  measuring  performance  against  an  agreed-upon  stan- 
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Figure  36.  Measurable  Quality  Improvement  Goals  Figure  37.  SPO  Monitor  Performance  to  Goals. 

Defined  by  Your  Organization 


dard  to  a  great  extent  (fig.  38).  An  equal  number  of  program  managers  and 
QA  personnel  Indicate  that  very  little  is  being  done  in  this  area.  More  than 
40%  of  the  program  directors  and  50%  of  the  program  managers  and  QA 
personnel  indicate  that  they  are  only  infrequently  comparing  planned  to 
actual  work  hours.  Furthermore,  more  than  30%  of  the  program  managers 
and  QA  personnel  indicate  that  they  are  not  using  this  data  at  all  (fig.  39). 

When  asked  to  what  extent  quality  personnel  focus  on  defect  prevention 
rather  than  detection,  the  respondents  show  significant  disagreement  on 
the  topic.  Only  20%  of  the  program  managers  and  35%  of  the  program 
directors  say  that  QA  focuses  to  a  great  extent  on  defect  prevention  (fig.  40). 
However,  55%  of  the  QA  personnel  say  their  primary  focus  is  on  prevention. 
Approximately  25%  of  the  program  managers  do  not  answer  this  question 
(do  not  know  where  the  focus  is). 

As  discussed  earlier,  approximately  40%  of  the  program  managers  and 
QA  personnel  think  very  little  resources  were  being  expended  to  improve 
quality  (fig.  4).  This  outcome  correlates  with  the  fact  that  more  than  40% 
think  that  the  SPO  uses  very  little  producibility  funding  to  reduce  risk  (fig. 
2).  Only  30%  of  the  program  managers  and  QA  personnel  and  41%  of  the 
program  directors  indicate  the  SPOs  are  using  quality  indicators  to  a  great 
extent  (fig.  41).  An  equal  number  also  say  they  are  using  indicators  very 
little. 

Quality  improvement  appears  to  be  an  important  program  objective  and 
is  one  that  contractors  seem  to  accept.  Although  the  SPOs  define  few 
measurable  QI  goals,  they  do,  to  some  extent,  monitor  performance  against 
the  goals  that  are  established.  Also,  the  SPOs  measure  contractors  perfor¬ 
mance  against  quality  standards  to  a  limited  degree  but  are  not  looking  at 
planned  versus  actual  work-hour  data.  Finally,  most  QA  people  use 
indicators  and  measures  of  quality  only  to  some  extent,  which  indicates 
that  the  quality  improvement  process  is  not  well  established  in  AFSC. 

On  the  average,  only  about  30%  of  the  work  force  perform  those  tasks 
that  can  greatly  increase  quality.  The  product  division  staffs  and  Head¬ 
quarters  AFSC  leadership  are  doing  very  little  to  help  the  SPOs  increase  the 
level  of  product  quality.  Warranties  and  award-fee  contracts  are  ways  to 
motivate  the  contractor  to  provide  better  quality  but  a  firm-fixed -price 
contract  will  not  do  so.  Also,  the  users  have  not  been  very  effective  in 
influencing  the  contractor  to  provide  better  quality.  In  general,  quality 
problems  have  not  affected  program  management  to  a  great  extent.  Al¬ 
though  some  design  and  manufacturing  problems  affect  cost  and  schedule 
to  some  extent,  they  have  little  impact  on  quality.  People  working  in  the 
SPO  are  not  using  the  QI  process  to  any  great  extent.  They  have  few 
measurable  quality  goals  to  work  with,  and  they  use  such  indicators  of 
quality  only  to  some  extent. 
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Figure  40.  Quality  Personnel  Focus  on  Defect  Figure  4 1 .  Use  of  Measures  and  Indicators  of  Quality. 

Prevention  versus  Detection. 


What  Is  the  Contractor  Doing  to  Affect  Quality? 


If  contractors  are  going  to  control  the  “hidden  factory”  costs  discussed 
earlier,  they  must  implement  a  total  quality  management  (TQM)  approach 
with  a  continuous  quality  improvement  program.  TQM  means  a  totally 
integrated  management  process  focused  on  quality  improvement  in  every 
organization  and  function,  not  just  for  hardware  in  the  production  phase. 
Under  a  TQM  approach,  everyone  is  responsible,  not  just  the  quality 
organization,  to  assure  a  quality  product,  whether  it  is  hardware,  software, 
or  paper  products. 

Several  survey  questions  focus  on  the  TQM  philosophy  and  one  specifi¬ 
cally  asks  if  the  contractor  is  using  this  approach.  TQM  is  not  defined  in 
the  questionnaire  and  approximately  30%  of  the  respondents  indicate  eit  her 
they  do  not  know  what  the  contractors  are  doing  or  that  they  believe  that 
the  contractors  are  doing  very  little  in  TQM  (fig.  42).  Nearly  40%  of  the 
program  directors  but  only  25%  of  the  program  managers  and  QA  personnel 
indicate  that  contractors  are  doing  a  lot  in  total  quality  management. 
However,  the  majority  of  the  respondents  think  quality  is  an  important 
objective  (fig.  3)  and  the  contractors  share  our  quality  objectives  (fig.  35). 

Since  TQM  is  not  defined,  several  survey  questions  ask  about  the 
contractor’s  involvement  with  TQM.  Approximately  55%  of  the  program 
directors,  but  only  about  30%  of  the  program  managers  and  QA  personnel, 
say  the  contractor’s  top  management  is  influencing  quality  to  a  great  extent 
(fig.  8).  Forty-five  percent  (45%)  of  the  program  directors  and  QA  personnel 
indicate  contractors  are  using  employee  motivation  to  some  extent  (fig.  43). 
Thirty  percent  (30%)  of  the  program  managers  fail  to  answer  this  question 
(do  not  know  what  the  contractor  is  doing).  Fifty-five  percent  (55%)  of  the 
program  directors  and  30%  of  the  program  managers  and  QA  personnel 
indicate  that  the  contractor  emphasizes  supplier's  quality  to  a  great  extent . 
But  30%  of  the  program  managers  indicate  they  do  not  know  what  the 
contractor  is  doing  and  30%  of  the  QA  personnel  indicate  they  either  do  not 
know  or  the  contractor  is  doing  very  little  to  assure  that  Ihe  supplier  is 
providing  quality  (figs.  19  and  20).  Forty  percent  (40%)  of  the  program 
directors  and  QA  personnel  indicate  that  the  contractor  has  a  preferred 
vendor  program  with  ratings  on  quality  and  schedule  (fig.  19).  However, 
only  25%  of  the  program  managers  agree  with  this  rating  and  another  25% 
indicate  they  do  not  know  if  the  contractor  has  a  vendor  rating  program. 

When  asked  whether  contractors  have  an  aetive  QI  program  based  upon 
continuous  evaluation,  40%  of  the  program  directors  and  QA  personnel  say 
that  the  contractor  uses  a  Ql  program  to  a  great  extent  (fig.  44).  Only  25% 
of  Ihe  program  managers  agree  with  this  assessment  while  another  25% 
say  they  do  not  know  if  the  contractor  has  a  Ql  program.  At  the  same  time. 
55%  of  the  program  managers  and  45%  of  the  QA  personnel  say  contractors 
have  done  very  little  in  conducting  subcontractor  QI  programs.  More  than 
25%  of  the  program  managers  do  not  answer  this  question  (do  not  know  if 
the  contractor  is  conducting  subcontractor  QI  programs). 
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The  use  of  subcontractor  QI  programs  as  a  means  of  achieving  better 
quality  is  not  being  used  adequately  by  the  contractors.  Forty-five  percent 
(45%)  of  the  QA  personnel  and  50%  of  the  program  managers  (25%  did  not 
answer)  say  design  teams  to  achieve  quality  improvement  are  being  used 
only  to  a  very  little  extent  (fig.  15).  Experts  in  quality  say  the  use  of 
statistical  process  controls  (SPC)  is  a  must  to  realize  true  improvements. 
Yet  approximately  55%  of  the  program  managers  and  QA  personnel  and 
40%  of  the  program  directors  say  SPC  techniques  are  being  used  only  to  a 
very  little  extent  (fig.  45).  Approximately  15%  of  the  QA  personnel.  25%  of 
the  program  directors,  and  35%  of  the  program  managers  do  not  answer 
this  question  (again,  indicating  they  do  not  know  the  answer).  When  asked 
to  what  extent  contractors  use  measures  or  indicators  of  quality,  19%  of 
the  QA  personnel,  27%  of  the  program  directors,  and  38%  of  the  program 
managers  do  not  know  if  the  contractor  is  using  various  indicators  of  quality 
(they  did  not  answer  the  question).  Also,  only  25%  of  the  program  managers 
and  QA  personnel  say  these  indicators  were  being  used  to  a  great  extent 
(fig.  46).  Almost  halfoft  he  program  managers,  36%  of  the  quality  assurance 
personnel,  and  24%  of  the  program  directors  indicate  either  they  do  not 
know  what  the  contractor  is  doing  or  that  the  contractor  is  doing  very  little 
to  improve  quality  in  the  areas  questioned.  About  40%  of  the  program 
directors  say  the  contractors  are  doing  a  lot  in  this  area,  but  only  25%  of 
the  program  managers  and  QA  personnel  reach  this  conclusion  (fig.  47). 

A  significant  number  of  people  in  the  work  force  indicate  they  do  not  know 
what  the  contractor  is  doing,  which  should  concern  Headquarters  AFSC. 
The  majority  of  the  respondents  say  contractors  are  using  a  TQM  approach 
only  to  some  extent,  not  a  strong  endorsement  for  TQM.  Only  about  one 
third  of  the  respondents  indicate  that  the  contractors  use  an  active  QI 
program  to  a  great  extent  and  less  than  20%  indicate  the  contractors  are 
using  statistical  process  control  to  a  great  extent.  Finally,  the  majority  of 
(he  work  force  indicates  that  quality  indicators  are  being  used  at  least  to 
some  extent,  but  a  significant  number  do  not  know  if  the  contractor  is  using 
these  indicators. 


Are  Indicators  of  Quality  Being  Used? 

One  of  the  problems  many  people  have  with  quality  is  the  perceived 
difficulty  in  defining  and  measuring  what  is  good  quality.  This  difficulty  is 
implied  in  the  discussion,  in  chapter  1  above,  of  Guaspari’s  book  I  Know  It 
XV} icri  l  See  It:  A  Modem  Fable  about  Quality.  It  is  also  expressed  in  research 
question  two  when  the  respondents  rank  the  nebulous  definition  of  quality 
as  the  filth  most  important  detractor  of  quality  and  productivity.  However, 
the  level  of  quality  can  be  associated  with  measurable  manufacturing  and 
design  parameters.  These  include  reliability  and  maintainability  data, 
production  yield  rates,  test  yields,  and  scrap,  rework  and  repair  rates, 
because  as  these  parameters  “go  bad"  so  goes  quality.  The  survey  ques- 
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Figure  44 .  Contractor  Instituted  an  Active  Figure  45.  Contractor  Using  Statistical  Process 

Quality  Improvement  Program  Control  Including  Vendors. 


tionnaire  addresses  to  what  extent  the  SPO  is  using  some  of  these  in¬ 
dicators. 

First,  respondents  are  asked  to  indicate  how  good  or  bad  they  think 
quality  is  In  AFSC  and  to  state  to  what  extent  a  few  specific  indicators  of 
quality  are  being  used.  When  asked  how  many  quality  of  conformance 
problems  had  significant  program  impacts  (schedule,  cost,  accident, 
publicity,  etc.)  during  the  past  year,  30%  say  they  have  no  significant  quality 
problems  while  40%  have  had  one  to  three  quality  problems  over  the  last 
year  that  had  major  impacts  on  the  program  (fig.  48).  Many  respondents 
(40%)  think  the  number  of  quality  problems  are  normal  for  a  program  of 
their  complexity  (fig.  30).  Approximately  60%  of  the  program  directors  and 
50%  of  the  program  managers  say  quality  has  very  little  impact  on  delivery 
schedules  (fig.  13).  However.  40%  of  the  QA  personnel  say  quality  affects 
delivery  schedules  to  a  great  extent.  When  asked  to  what  extent  manufac¬ 
turing  and  engineering  errors  cause  problems,  approximately  50%  of  the 
respondents  say  they  do  cause  some  problems  in  managing  the  program 
(figs.  31  and  32).  Each  of  these  items  is  a  general  indicator  of  quality  and 
can  and  should  be  monitored  for  trends  and  repeated  occurrences  of  the 
same  problem.  For  example,  a  trend  analysis  of  manufacturing  and 
engineering  errors  can  indicate  if  quality  of  design  is  improving  or  stabiliz¬ 
ing.  Any  quality  problem  causing  major  program  impact  should  be  an 
obvious  indication  of  poor  quality  and  a  lack  of  control. 

This  research  also  seeks  to  determine  to  what  extent  the  work  force  is 
using  five  specific  indicators  of  quality.  One  strong  indicator  of  quality  is 
the  commitment  to  design  quality  into  the  product.  Sixty  percent  (60%)  of 
the  program  directors  but  only  about  30%  of  the  program  managers  and 
QA  personnel  indicate  they  are  designing  quality  into  the  product  (fig.  1). 
When  asked  to  what  extent  field  performance  data  is  used  to  indicate  or 
measure  quality,  almost  50%  of  the  respondents  indicate  they  use  field  data 
to  a  great  extent  (fig.  49).  However,  15%  of  the  program  managers  and  18% 
of  the  program  directors  fail  to  answer  this  question.  Forty  percent  (40%) 
of  the  program  directors  and  50%  of  the  QA  personnel  and  program 
directors  indicate  that  they  seldom  compared  planned  to  actual  work-hour 
data  (fig.  41).  Approximately  50%  of  the  program  directors  and  QA  person¬ 
nel  indicate  they  are  monitoring  defects  and  workmanship  data  (fig.  6). 
Only  30%  of  the  program  managers  gave  this  response  and  40%  say  they 
are  monitoring  this  data  to  a  very  little  extent.  Fifty  percent  (50%)  of  the 
program  directors  and  QA  personnel  and  65%  of  the  program  managers 
indicate  that  they  use  subcontractor  yield  data  to  a  very  little  extent  (fig. 
18). 

The  respondents  are  also  asked  to  assess  to  what  extent  they  think  the 
contractor  is  using  indicators  of  quality.  Approximately  40%  of  (he  QA 
personnel  and  30%  of  the  program  directors  and  managers  say  cost, 
schedule,  performance,  and  reliability  affect  design  and  manufacturing 
problems  to  a  great  extent  (fig.  33).  When  asked  to  what  extent  a  contractor 
uses  product  return  rates  (cost  to  rework)  as  an  indicator  of  quality. 
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48.  Number  of  Significant  Quality  Problems  Figure  49.  Using  Field  Performance  Data. 


approximately  40%  of  the  program  directors  and  QA  personnel  and  60%  of 
the  program  managers  say  they  either  do  not  know  or  say  the  contractor  is 
using  this  indicator  infrequently  (fig.  50).  Twenty  percent  (20%)  of  the  QA 
personnel,  27%  of  the  program  directors,  and  4 1  %  of  the  program  managers 
do  not  know  if  the  contractor  uses  this  indicator  (they  did  not  answer  the 
question).  Approximately  40%  of  the  program  directors  and  QA  personnel 
and  60%  of  the  program  managers  say  either  that  they  do  not  know  or  that 
the  contractor  is  doing  very  little  to  standardize  manufacturing  processes 
(fig.  51).  The  number  of  respondents  who  do  not  answer  is  very  similar  to 
the  previous  question.  When  asked  whether  the  contractor  is  using  produc¬ 
tion  yield  rates  as  an  indicator  of  quality,  again  approximately  40%  of  the 
program  directors  and  QA  personnel  and  55%  of  the  program  managers 
either  do  not  answer  or  say  production  yield  rates  are  used  to  a  very  little 
extent  (fig.  52). 

Only  30%  of  the  program  managers  and  QA  personnel  indicate  the 
contractors  are  using  MRB/QDR  action  effectiveness  (repeats)  as  an  in¬ 
dicator  of  quality  to  a  great  extent .  Howeve» ,  more  than  50%  of  the  program 
directors  feel  that  contractors  are  using  this  data  (fig.  5).  A  significant 
number,  more  than  20%.  of  the  respondents  do  not  answer  this  question. 
When  asked  whether  they  are  using  accept-reject  rates  at  work  centers, 
approximately  30%  of  the  program  directors  and  QA  personnel  and  50%  of 
the  program  managers  either  do  not  answer  or  say  this  indicator  is  used  to 
a  very  little  extent  (fig.  53). 

One  purpose  of  this  research  is  to  assess  to  what  extent  the  acquisition 
work  force  in  the  SPO  and  the  contractor  are  using  indicators  or  measures 
of  quality.  Although  a  significant  number  of  the  respondents,  approximate¬ 
ly  70%,  indicate  they  are  using  quality  indicators  at  least  to  some  extent, 
only  30-40%  say  they  are  using  them  to  a  great  extent  (fig.  42).  Ap¬ 
proximately  55%  of  the  respondents  indicate  that  the  contractor  is  using 
quality  indicators  to  some  extent ,  but  again  only  30-40%  say  they  are  using 
them  to  a  great  extent  (fig.  46).  The  significant  number  of  respondents  who 
do  not  answer  (because  they  do  not  know  the  answer)  should  be  of  concern 
to  Headquarters  AFSC. 

Are  SPO  Personnel  Using  Cost  of  Quality 
Data  as  a  Program  Management  Tool? 

Another  aim  of  this  research  is  to  determine  to  what  extent  the  work  force 
knows  what  the  cost  of  quality  is  on  their  program  and  to  determine  if  they 
are  reviewing  that  data.  This  data  is  important  since  these  costs  add  no 
value  to  the  product  and  are  estimated  by  experts  in  industry  and  govern¬ 
ment  to  be  as  high  as  35%  of  sales.  These  costs  offer  a  tremendous 
opportunity  for  increasing  productivity  and  reducing  costs  to  those  firms 
that  have  begun  to  use  cost  of  quality  concepts  as  a  management  tool.  As 
indicated  earlier,  most  respondents  say  they  have  had  one  to  three  sig¬ 
nificant  quality  problems  over  the  last  year  (fig.  48).  These  types  of  problems 
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Figure  52.  Production  Yield  Rates  as  Figure  53.  Accept  or  Reject  Rate  at  Work  Centers  as 

a  Quality  Indicator.  a  Quality  Indicator. 


usually  Indicate  serious  quality  problems  and  increase  costs  to  the  govern- 
ment  and  contractor.  When  asked  how  often  they  review  user-complaint 
data  and  trends,  a  significant  number  of  the  program  management  and  QA 
personnel  say  they  never  review  user-complaint  trends.  However,  the 
majority  of  the  respondents  indicate  they  do  review  user  data  (fig.  54)  and 
do  use  field  performance  data  (fig.  49).  Many  respondents — 20%  of  the 
program  directors  and  QA  personnel  and  50%  of  the  program  managers — 
indicate  they  never  review  contractor's  scrap,  rework,  repair,  and  retest  data 
(fig.  11).  This  finding  is  alarming  because  in  all  likelihood  the  contract 
requires  this  data.  When  asked  how  often  they  personally  review 
contractor’s  cost  of  quality  (failure  costs  +  routine  inspection  +  prevention 
costs)  data  or  trends,  approximately  40%  of  the  program  directors  and  QA 
personnel  and  60%  of  the  program  managers  say  they  never  review  this 
data  (fig.  12).  (As  a  result  of  this  high  negative  response,  the  confidence  in 
the  following  data  is  very  low.)  Approximately  60%  of  the  respondents 
indicate  that  the  cost  of  quality  is  less  than  15%  of  their  contract  value,  and 
less  than  10%  say  it  is  greater  than  25%  (fig.  55). 

Next,  the  respondents  are  asked  to  break  down  the  three  components  of 
the  cost  of  quality  and  if  they  do  not  know  to  leave  the  question  blank.  The 
component  costs  are  failure,  inspection,  and  prevention  costs.  The  largest 
single  response,  approximately  60%.  indicate  that  they  do  not  know  what 
the  individual  costs  are.  A  significant  number,  about  25%.  say  the  total 
cost  of  quality  is  divided  approximately  equally  between  the  three  com¬ 
ponents  (fig.  56).  However,  literature  indicates  that  most  of  the  companies 
using  cost  of  quality  concepts  have  not  measured  all  components  and  have 
been  focusing  on  failure  costs  because  they  are  easier  to  measure.  Similar¬ 
ly,  when  asked  how  much  confidence  they  have  in  their  cost  of  quality 
estimate,  approximately  50%  of  the  respondents  do  not  answer  (zero 
confidence  in  their  estimate);  many  others  indicate  that  they  have  less  than 
25%  confidence  in  their  estimate  (fig.  57). 

The  data  indicates  that  approximately  one  half  of  the  work  force  never 
review  scrap,  rework,  and  repair  data  and  the  cost  to  the  government  for 
these  items.  Even  if  cost  of  quality  data  is  being  reviewed,  it  is  not  likely 
that  the  SPO  personnel  can  separate  this  data  into  its  components:  failure, 
inspection,  and  prevention  costs.  Finally,  the  data  indicates  that  the  many 
members  of  the  work  force  do  not  know  the  cost  of  quality  for  their  program. 
The  literature  indicates  that  a  major  obstacle  in  using  cost  of  quality  data 
is  getting  supervisors  to  believe  that  these  are  true  costs  that  can  be  reduced 
and  avoided.2 


Summary 

This  chapter  summarizes  the  empirical  findings  from  survey  question¬ 
naires  sent  to  acquisition  personnel  at  Air  Force  Systems  Command's  five 
product  divisions.  The  objective  of  the  questionnaires  is  to  determine  if 
AFSC  acquisition  personnel  are  aware  of  and  using  those  indicators  of 
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Figure  56.  Estimate  of  the  Cost  of  Quality  Components:  Figure  57.  Confidence  in  Cost  of  Quality  Estimate. 

Failure,  Inspection,  and  Prevention  Costs. 


quality  that  more  effectively  identify  nonconformances  and  are  applying 
techniques  to  reduce  incidences  of  nonconformance. 


Notes 

1.  “Accountable  Contract  Management.  AFC'Mi)  Review."  briefing  to  the  AFSC  com¬ 
mander,  December  1984. 

2.  iTiomas  C.  Gibson.  "Hie  Total  Quality  Management  Resource."  Quality  Progress. 
November  1987.  63. 


CHAPTER  5 


Summary  and 
Recommendations 


The  acquisition  work  force  must  realize  that  often  the  Defense  Depart¬ 
ment  is  paying  too  much  for  the  products  and  weapon  systems  it  acquires. 
Tms  work  force  must  take  appropriate  action  to  reduce  the  cost  of  those 
products  and  systems.  Some  critics  have  estimated  that  the  Defense 
Department  has  been  paying  a  significant  portion  of  its  procurement  budget 
(20-30%)  to  contractors  for  costs  that  add  no  value  to  the  item.  Until  very 
recently,  the  Defense  Department  and  the  aerospace  industry  have  paid 
little  attention  to  these  costs,  termed  as  the  hiddenfactonj  by  General  Marsh 
in  1982.  But  this  situation  is  changing  now  that  high-level  DOD  personnel 
are  becoming  more  aware  of  the  significance  of  these  costs  and  now  that 
DOD  is  experiencing  a  reduction  in  its  budget.  The  objective  of  this  study 
is  to  determine  if  Air  Force  Systems  Command  (AFSC)  acquisition  person¬ 
nel — the  program  directors,  program  managers,  and  quality  assurance  (QA) 
personnel  at  the  five  AFSC  product  divisions — are  aware  of  and  using  those 
techniques  that  more  effectively  identify  nonconformances  and  are  applying 
techniques  to  reduce  incidences  of  nonconformance.  This  researcher  uses 
a  questionnaire  to  determine  to  what  extent  the  work  force  really  under¬ 
stands  what  these  costs  are  and  to  determine  the  extent  to  which  it  is  using 
this  data  to  manage  programs. 

The  overall  response  rate  for  the  questionnaire  was  about  35%.  The  rate 
of  response  can  be  indicative  of  the  level  of  interest  in  the  topic  on  the  part 
of  those  surveyed.  The  response  in  this  study  can  provide  a  statistically 
accurate  estimate  of  the  population  with  a  95%  confidence  level  and  an 
error  rate  of  less  than  10%,  which  is  considered  an  acceptable  level  for  this 
type  of  data.  The  response  for  each  subpopulation  (program  directors, 
program  managers,  and  quality  assurance  personnel)  appears  to  be  repre¬ 
sentative  of  each  respective  group.  The  “typical"  program  director  is  a 
colonel  with  more  than  10  years  of  acquisition  experience.  Ninety-four 
percent  (94%)  manage  programs  that  are  in  the  full-scale  development  or 
production  phases.  The  “typical”  program  manager  is  a  captain  with  less 
than  three  years  of  acquisition  experience.  Only  16%  work  on  systems  in 
the  conceptual  or  demonstration-validation  phases.  The  “typical"  QA 
civilian  is  a  GM- 12  engineer  with  10  years  of  acquisition  experience  and  the 
“typical"  QA  militaiy  officer  is  a  first  lieutenant  with  an  engineering  degree. 
Only  16%  of  the  QA  personnel  work  on  programs  in  the  conceptual  or 
demonstrat  ion- validation  phases. 
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The  data  strongly  indicates  that  there  is  a  lack  of  communication  among 
the  acquisition  work  force.  The  findings  graphically  show  the  magnitude 
of  how  the  three  subpopulations  answer  various  questions  differently.  A 
one-way  analysis  of  variance  test  using  a  95%  confidence  level  reveals  that 
all  three  subpopuialions  agree  on  only  17  of  59  possible  questions.  The 
program  directors  and  QA  personnel  agree  on  24  of  59  questions.  The 
program  managers  and  QA  personnel  agree  on  40  of  59  questions:  and  the 
progiam  din  e  tors  and  program  managers  agree  on  19  of  59  questions. 
Additionally,  the  program  directors  consistently  answer  questions  more 
positively  than  the  other  two  subpopulations.  This  outcome  may  result 
because  the  directors  are  more  informed,  experienced,  or  optimistic  in¬ 
dividuals.  or  because  the  program  managers  and  QA  personnel  know  the 
problems  of  the  systems,  yet  do  not  report  the  bad  news  up  to  the  next  layer 
of  the  bureaucracy. 

A  significant  problem  also  exists  when  15-20%  of  the  QA  work  force 
cannot  answer  many  basic  questions  concerning  what  a  contractor  is  doing 
with  respect  to  quality  improvement.  This  finding  needs  to  be  investigated 
further  by  Headquarters  AFSC.  The  problem  may  be  a  lack  of  focus  on  the 
correct  tasks,  but  it  is  probably  a  lack  of  available  manpower  and  training. 
Approximately  15-20%  of  the  program  directors  and  an  even  greater 
number  of  program  managers  (as  high  as  40%)  do  not  answer  many  of  the 
survey  questions  addressing  the  subcontractor.  This  outcome  is  surprising 
■  onsid-Tine  ! ! .  a9e.,ijon  Headquarters  AFS( '  has  put  on  subcontract 
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Hi.  y  iii-  ■'  •  :der stand  t  ha  ■  ■  is  an ■  oil en  too  high  and  that  the  Air  Force 
and  l<«n  .<o •  alien  not  gt  1 1 ii  quality  pioducts.  This  fact  is  true  regardless 
ot  the  acquisition  phase.  type  of  contract,  or  even  if  the  product  is  a 
ronmiei eial  oil  the  shell  item. 


flow  llav  v  t.-:l  {Studies  Addressed  This  Topic? 

Quality  assn i am  e  has  been  ot  continuing  concern  to  the  Air  Force  the 
last  ci >i : pie  of  decades.  During  the  last  10  years,  AFSC  has  conducted 
seveial  major  reviews  ot  quality  assurance  (QA).  In  the  previous  15  years, 
t  he  Defense  I  )epart  incut  and  Air  Force  have  done  some  20  studies  on  quality 
assuram  e  Seveial  significant  changes  to  the  quality  assurance  function 
have  ii -suited  fiom  the  i e<  ommcndal ions  of  these  studies.  The  current 
st  udv  i  *  h  lies  many  ol  ( lie  same  conclusions  as  several  of  the  past  studies. 
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These  include  a  need  for  Headquarters  AFSC  to  establish  a  clear  and  firm 
policy  of  not  accepting  poor  quality;  the  need  to  use  scrap,  rework,  and 
repair  levels  with  cost  data  to  manage  contractor  performance;  the  need  to 
use  more  positive  incentives  to  motivate  the  contractor  to  deliver  better 
quality;  and  the  need  for  better  education  and  training  for  QA  personnel. 
These  findings  and  recommendations  have  been  around  for  at  least  1 0  years 
and  some  even  longer. 


What  Are  the  Most  Important 
Factors  Affecting  Quality? 

The  survey  asks  the  acquisition  work  force  to  indicate  what  are  the  most 
important  factors  affecting  quality.  One  question  asks  the  respondents  to 
rank  the  top  contributors  to  quality  and  another  asks  them  to  rank  the  top 
detractors  from  quality.  The  respondents  identify  the  three  top  contributors 
to  a  quality  product  as  early  inputs  to  the  design  (producibility.  manutae- 
tu ring,  and  quality),  commitment  to  not  accepting  poor  quality,  and  pro¬ 
gram  stability.  They  list  the  top  three  detractors  to  quality  as  unrealistic 
program  schedules,  lack  of  cooperation  among  personnel,  and  program 
instability.  They  indicate  that  quality  is  an  important  program  objective 
and  that  they  are  using  a  development  process  that  designs  quality  into  the 
product  to  some  extent.  However,  in  answers  to  several  other  questions, 
the  work  force  indicates  that  it  is  accomplishing  very  few  of  the  important 
contributors  to  a  quality  product  to  any  real  extent.  These  include 
producibility  funding,  educational  programs,  team  building,  using  design 
teams,  and  establishing  and  monitoring  quality  goals. 

Secretary  of  Defense  Frank  Carlucci  has  recognized  and  endorsed  the 
need  for  a  “cultural  change  and  fundamental  redefinition  to  how  our 
institutions  deal  with  quality."  Much  of  the  data  presented  in  chapter  4 
indicates  that  the  work  force  does  not  understand  this  new  definition  of 
quality  (total  quality  management).  The  high  percentage  of  questions  that 
were  answered  “to  some  extent”  indicates  a  lack  either  of  commitment  or 
understanding  of  TQM  and  the  continuous  quality  improvement  process. 
For  AFSC  to  achieve  the  quality  revolution  that  General  Marsh  referred  to 
in  1982,  70-80%  of  the  responses  should  have  been  in  the  “great  to  very 
great  extent"  categories. 

What  Is  the  System  Program  Office 
Doing  to  Affect  Quality? 

Another  focus  of  this  research  is  to  determine  the  effectiveness  of  the  QA 
organization  in  the  system  program  office  (SFO).  the  effectiveness  of  motiva¬ 
tional  techniques  to  improve  quality,  the  degree  to  which  quality  problems 
are  affecting  program  management,  and  the  degree  to  which  the  Sl'O  is 
using  the  continuous  quality  improvement  process.  The  QA  work  force  is 
only  somewhat  effective  in  contributing  to  high  quality  products.  The 
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responses  indicate  that  this  shortcoming  is  probably  more  a  function  of 
manning  and  training  than  of  the  relative  importance  of  quality  in  the 
organization.  The  QA  personnel  disagree  with  the  program  directors  and 
managers  as  to  the  effectiveness  of  various  methods  for  influencing  the 
contractor  to  deliver  better  quality  products.  Significantly  more  program 
directors  and  managers  indicate  that  good  business  practices  and  user 
involvement  in  the  acquisition  process  can  influence  the  contractor  to 
provide  better  quality.  There  is  closer  agreement  on  the  effectiveness  of 
contract  requirements,  warranties,  and  award  fees  and  iiie  lack  of  effective¬ 
ness  of  firm- fixed -price  contracts.  Most  respondents  indicate  that  design, 
engineering,  and  manufacturing  errors  do  cause  some  problems  in  manag¬ 
ing  their  program,  but  they  feel  that  the  number  of  quality  problems  is 
normal  for  complex  programs.  Approximately  50%  of  the  respondents  state 
that  they  have  three  or  more  significant  quality  problems  a  year.  Hence, 
they  expect  to  have  some  quality  problems,  and  they  feel  that  it  is  not 
unusual  to  face  several  significant  quality  problems.  This  altitude  must 
change  because  the  Air  Force  and  AFSC  can  no  longer  accept  the  costs 
associated  with  acceptance  of  low  levels  of  quality.  Although  the  majority 
indicate  that  a  contractor's  performance  is  measured  against  quality  stan¬ 
dards  at  least  to  some  extent,  the  responses  indicate  that  the  SPO  has 
established  few  measurable  quality  improvement  goals  and  the  comparison 
of  the  contractor’s  planned  versus  actual  work-hour  data  is  being  done  to 
a  very  little  extent.  This  data  shows  that  those  in  the  acquisition  community 
are  not  universally  practicing  the  continuous  quality  improvement  process. 


What  Is  the  Contractor  Doing  to  Affect  Quality? 

A  significant  part  of  the  work  force  is  not  aware  of  what  the  contractor  is 
doing  with  the  quality  improvement  (QI)  process  or  total  quality  manage¬ 
ment  (TQM).  This  void  is  particularly  evident  in  the  area  of  what  the  prime 
contractor  is  doing  with  subcontractors.  Most  respondents  say  contractors 
are  using  a  TQM  approach  only  to  some  extent  and  only  a  third  say  an  active 
QI  program  has  been  instituted  to  a  great  extent.  Statistical  process  control 
(SPC)  is  a  veiy  important  tool  for  an  active  QI  process.  However,  only  20% 
say  contractors  are  using  SPC  to  a  great  extent. 

Are  Indicators  of  Quality  Being  Used? 

Approximately  70%  of  the  respondents  indicate  that  the  SPOs  use  some 
quality  indicators  at  least  to  some  extent,  but  only  30-40%  use  them  to  a 
great  extent.  Of  those  indicators  listed  in  the  survey,  using  field  perfor¬ 
mance  data  is  the  most  often  selected  item  with  almost  5G%  indicating  they 
are  using  this  data  to  a  great  extent.  Over  50%  indicate  they  are  using 
subcontractor  yield  data  to  a  very  little  extent  or  not  at  all.  When  asked  to 
what  extent  the  contractors  use  quality  indicators,  a  significant  number  of 
respondents  do  not  know  if  contractors  are  using  quality  indicators  at  all. 
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Only  about  55%  of  the  respondents  indicate  that  the  contractor  is  using 
quality  indicators  to  some  extent  or  more,  the  rest  either  do  not  know  or 
indicate  very  little  to  not  at  all. 


Are  SPO  Personnel  Using  Cost  of  Quality  Data 
as  a  Program  Management  Tool? 

Because  of  the  importance  of  managing  the  cost  of  quality,  several  survey 
questions  focus  on  this  indicator  of  quality.  A  significant  number  of  the 
respondents  (20%  of  the  program  directors  and  QA  personnel  and  50%  of 
the  program  managers)  indicate  they  never  review  the  major  components 
of  this  cost:  scrap,  rework,  and  repair  levels.  An  even  larger  number  (45% 
of  program  directors  and  QA  personnel  and  65%  of  the  program  managers) 
say  they  never  review  the  cost  associated  with  these  items.  If  AFSC  is  to 
reduce  “hidden  factories,”  it  must  recognize  scrap,  rework,  and  repair  as 
true  costs  and  it  must  recognize  that  the  magnitude  of  those  costs  is  a  major 
obstacle  to  achieving  quality  improvement  (QI). 


Recommendations 

AFSC  needs  to  demonstrate  clearly  its  commitment  to  total  quality 
management  and  provide  strong  leadership  to  ensure  that  TQM  is  imple¬ 
mented.  Several  recent  examples  indicate  that  both  DOD  and  AFSC  are 
moving  toward  a  commitment  to  TQM.  First,  in  April  1988,  Daniel  S.  Rak, 
deputy  assistant  secretary  of  defense  (acquisition  management  and  policy), 
solicited  comments  from  AFSC,  among  others,  on  three  new  DOD  docu¬ 
ments:  DOD  Total  Quality  Management  Master  Plan ,  DODD  5000 .XX  ( Total 
Quality  Management),  and  DOD1  5000.YY  ( Quality  Program).  Rak’s  letter 
suggests  that  these  documents  on  TQM  will  help  bring  about  a  cultural 
change  and  redefine  how  the  acquisition  community  deals  with  quality. 
Second,  the  AFSC  commander,  Gen  Bernard  P.  Randolph,  issued  a  policy 
letter  on  reliability,  maintainability,  and  producibility  (RM&P)  on  15  April 
1988.  This  letter  states  that  RM&P  are  key  building  blocks  of  TQM  and 
continuous  QI  and  instructs  program  directors  to  establish  measurable 
RM&P  objectives  and  to  show  how  their  project  is  reaching  those  objectives. 
Third,  in  a  letter  on  total  quality  management  in  AFSC  dated  12  May  1988. 
to  which  Secretary  Carlucci's  memorandum  on  TQM  principles  was  at¬ 
tached,  General  Randolph  states  his  commitment  to  make  TQM  “a  way  of 
life”  in  AFSC.  He  indicates  he  is  sending  40  select,  senior-level  people  from 
the  headquarters  stall  to  a  four-day  Deming  Quality  and  Productivity 
Seminar  on  TQM.  He  also  charges  each  field  commander  and  staff  to  seek 
similar  training. 

Headquarters  AFSC  must  show  that  its  commitment  to  and  leadership 
of  the  TQM  effort  does  not  change  with  personnel.  Ttie  leadership’s  strong 
commitment  to  quality  must  be  delivered  to  the  product  divisions  and  Air 
Force  Contract  Management  Division  (AFCMD)  work  force  by  actions,  not 
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just  words.  The  number  of  p  rt cnnel  who  attend  TQM  seminars  and  their 
actions  afterwards  mav  beeoi a-  in  indicator  ot  t  his  commitment .  Requiring 
program  directors  to  establish  goals  lor  t  he  various  quality  indicators,  such 
as  t  he  RM&P  object  ives,  and  to  give  briefings  on  t  heir  success  in  achieving 
them  at  program  review  would  indicate  command  interest  and  commit¬ 
ment  h  .  improvement  T.:  us  commit  ment  to  quality  improvement  can  also 
be  shown  1  >v  increasing  i  he  to  uulier  ( >;  QA  personnel  in  t  lie  buying  divisii  ms 
Slots  ran  betaken  from  i  1  ■  Air  lot  ce  ( 'ont ract  Management  1  )i  vision  quality 
In  net  ion  ot  t  he  1  )eiense  I  depart  men)  can  show  its  commit  ment  by  aut  horiz 
ing  new  slots.  However,  increasing  the  number  of  QA  slots  without  training 
would  have  minimal  effect . 

AFSC  needs  to  increase  education  and  training  in  total  quality  manage 
ment.  Almost  even.'  study  in  the  past  20  years  has  reached  the  same 
conclusion,  but  little  has  been  done  io  '  orreot  this  serious  deficiency.  The 
1980  DOI)  Authorization  Act  ([>[.  99  149,  sec.  1024)  requires  mandatory 
training  for  person n<  !  responsibic  for  assuring  quality  in  contractor 
facilities.  Phis  provision  can  very  easily  be  interpreted  to  mean  all  aequisi 
lion  personnel  Howey  .  tin*  Air  force  appanmtlv  ii.is  interpreted  it  to 
mean  nub,  ■.  juality  assurance  personnel  located  at  t  he  coni  ract or  s  lac il it  ies. 
Addil ionallv.  at  least  one  paoduet  division  lias  interpreted  the  law  as 
requiring  t  raining  only  for  new  hires  below  the  grade  of  GS- 11  or  -  1 2.  since 
this  grade  is  hired  as  fully  qualified." 

T  his  study  indicates  Dial  most  of  Die  acquisition  work  force  wauls  the 
training  and  education  s, i  that  they  will  be  more  effective  on  their  job.  but 
they  are  get  ting  lit  t  le  support  from  stalls  and  higher  headquarters.  In  fiscal 
vent  I  988  on  I  v  91  !  raining  slots  were  allocated  to  AFSC  for  QA  training  out 
of  a  loial  AFSf  ’  QA  work  Ion  >  ;4  approximately  1  .800,  Flic  only  mandatory 
training  that  mi  lit  a  rv  (.4  A  p»i  •-oimel  receive  is  that  requ  ired  1  >  v  the  law.  and 
that  may  not  apply  to  those  corking  in  the  SPOs.  Also,  there  is  no  local 
point  at  IT  nuim  u  a  iw  AfS( '  •<*  advo*  » t  *  a  training  program  for  the  mi  lit  an¬ 
as  111*'!!-  :>  I"!  1  •  1  v  i  1 1  a  n  s .  I  leadquai  in  s  AFSC  is  developing  a  mandatory 
civilian  liammi,  pu-  m  tmi  no  one  knows  how  much  support  il  has  at  Air 
Force  or  I )( )l )  levi  -l  !  Visit  ivc  and  visible  act  ion  on  I  his  f  ront  is  cssent  ial. 

The  speeiiie  reeoimiiendations  for  (raining  ai  >• 

I.  The  program  directors.  managers,  and  enginernng  weak  force  must 
he  ini  r  odui cd  t o  TQM  Hv  l,u  l  lie  most  -uymlii  11  it  impact  to  qualit y  can  be 
mack'  by  t  hese  individuals,  and  therefore  they  should  In'  Ihe  major  locus  of 
AFSC  quality  training.  At  Headquarters  AFSC.  the  deputy  chief  HI  stall, 
plod  net  assurance  and  acquisition  logistics  (AFSC/PI.).  should  review  the 
content  of  acquisition  courses,  such  as  those  taught  hv  the  Air  Force 
Institute  o|  Technology  (AI  I P)  and  the'  De  fense  Systems  Management 
College  ll)SM('),  and  ensure  that  the  curricula  include  material  on  TQM. 
T  he  stall  at  AFSC/PI.  should  per  iodic  ally  icview  < hese  curricula  to  make 
cc-ilam  that  the  course  material  is  .  ippr  opi  ial  e 
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2.  A  training  program  must  bo  established  for  the  military  and  eivilian 
QA  work  foree.  This  program  should  be  the  responsibility  of  Headtjuarters 
AFSC,  deputy  ehief  of  stall,  produet  assurance  and  acquisition  logistics. 
The  unique  requirements  of  each  product  division  and  Air  Force  Contract 
Management  Division  should  be  considered  in  developing  this  training 
program.  If  an  adequate  training  program  is  not  established  for  the 
military,  then  the  Air  Force  n!;o«:ki  consider  reassigning  these  officers  to 
duties  (hey  have  been  trained  to  perform. 

Additionally,  AFSC  needs  to  use  more  positive  incentives  to  motivate  the 
contractor  to  deliver  higher  quality  and  less  expensive  products.  Several 
previous  studies  make  this  same  recommendation.  The  work  force  has 
consistently  endorsed  the  use  of  incentives  for  many  years  and  the  majority 
indicate  in  this  study  that  both  warranties  and  award  fees  would  motivate 
their  contractor  to  deliver  better  quality  products. 

The  specific  recommendation  is  that  I  leadquarters  AFSC,  deputy  chief  of 
stall,  product  assurance  and  acquisition  logistics,  and  the  deputy  chief  of 
stall',  contracting  (AFSC/PK).  develop  an  award-fee  plan  for  quality  as¬ 
surance.  In  developing  this  plan,  these  offices  should  seek  inputs  from  the 
functional  areas  at  the  product  divisions  io  ensure  applicability  to  the 
various  products  and  acquisition  phases.  An  example  of  an  award-fee  plan 
for  quality  was  developed  in  the  study  Contract  Incentives  for  Product 
Quality,  but  I  suspect  it  has  rarely  been  used.  Once  this  plan  is  developed, 
the  work  force  will  have  to  lie  trained  to  implement  it. 

Last,  as  has  been  recommended  in  several  previous  studies,  in  order  for 
manufacturers  to  really  know  what  quality  costs,  they  must  start  collecting 
and  reviewing  data  that  shows  the  costs  of  mistakes  in  design,  manufac¬ 
turing.  and  support.  Since  at  least  as  far  back  as  the  1977  quality  study, 
the  Air  Force  has  recognized  the  need  to  obtain  measurable  data  on  quality, 
including  both  quality  performance  and  cost  data,  'file  results  presented 
in  this  study  show  clearly  that  little  performance  or  cost  data  is  being 
reviewed  by  SPO  personnel.  This  finding  is  particularly  disconcerting  since 
DOD  Directive  4155.1,  Quality  Program,  and  Air  Force  Regulation  74-1, 
Quality  Assurance  I^ogram,  require  a  quality  history  on  the  contractor  to 
be  maintained  so  that  contracts  will  not  be  awarded  to  contractors  who 
have  a  history  of  delivering  poor  quality  products. 

fhe  specific  recommendations  are  as  follows: 

1 .  Headquarters  AFSC  should  require  that  the  SPOs  and  AFCMD  obtain 
contractor  quality  data,  such  as  scrap,  rework,  repair,  and  retest  data,  as 
well  as  the  cost  associated  with  these  items.  Goals  should  then  be 
established  to  try  to  reduce  these  items,  fhe  SPOs  and  AFCMD  should  be 
working  with  the  product  division  staffs.  Headquarters  AFSC.  and  the 
contractor  to  ensure  that  the  data  is  in  a  usable  format  and  used  as  a 
program  management  tool. 
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2.  To  show  commitment  to  TQM  and  continuous  quality  improvement. 
Headquarters  AFSC  should  require  that  this  data  be  briefed  at  the  com¬ 
mand-level  program  reviews. 

3.  Headquarters  AFSC.  Deputy  Chief  of  Staff,  Product  Assurance  and 
Acquisition  Logistics,  should  develop  an  objective  methodology  to  evaluate 
and  monitor  contractor  quality  performance.  This  DOD  and  Air  Force 
requirement  has  historically  been  ignored  by  the  command  because  the 
work  force  has  not  understood  how  to  do  it.  Appendix  A  shows  an  example 
developed  in  a  previous  study  that,  although  having  a  subjective  rating 
scale,  would  provide  a  uniform  format  for  assessing  a  contractor’s  perfor¬ 
mance.  This  example  would  be  an  acceptable  format  if  the  work  force  were 
given  instructions  on  how  to  standardize  ratings.  Headquarters  AFSC 
should  then  maintain  summary  data  on  each  contractor  and  subcontractor. 

Concluding  Observations 

The  Defense  Department  and  Headquarters  AFSC  have  recently  shown 
they  have  an  Interest  in  improving  quality  in  order  to  reduce  costs.  Whether 
the  true  commitment  is  there  to  make  this  cultural  change  will  be  seen  in 
time.  The  work  force  indicates  in  this  study  that  the  leadership  at  the 
product  divisions  and  headquarters  has  done  little  to  help  obtain  better 
quality.  The  work  force  will  be  looking  for  the  deeds,  such  as  the  resources, 
training,  and  the  right  decisions,  coming  from  Headquarters  AFSC. 
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APPENDIX  A 


Contractor  Quality 
Performance  Evaluation  Summary 


A  suggested  format,  included  as  Attachment  1  to  this  Appendix,  is 
designed  to  provide  a  unifonn  format  for  assessing  the  contractor’s  perfor¬ 
mance  and  to  provide  data  that  can  be  used  to  fulfill  the  requirements  of 
policy  guidance  that  requires  evaluation  and  utilization  of  contractor 
performance  data.  The  recommended  evaluation  summary  is  divided  into 
three  categories:  Quality  of  Quality  Management  Systems.  Quality  of 
Product  Conformance  and  Quality  of  Design.  The  assessments  are,  to  a 
significant  degree,  judgmental.  The  absolute  data  that  is  included  will 
assist  in  providing  a  rationale  for  the  assessments.  That  data  will  also,  if 
collected  over  a  period  of  time,  and  over  a  range  of  products,  provide  a  basis 
of  “normal"  data  which  can  be  used  to  successfully  evaluate  and  efficiently 
administer  a  particular  contract. 

In  order  to  assure  a  thorough  understanding  of  the  evaluation  process, 
the  evaluator  must  prepare  a  written  description  or  rationale  of  the  factors 
that  caused  the  assignment  of  the  rating  which  is  then  multiplied  by  a  value 
factor  to  arrive  at  an  effectiveness  rating.  It  is  not  possible  to  provide 
standard  rationale  in  the  format  due  to  the  diversity  in  products  and 
contracts.  Accordingly,  each  evaluator  should  discuss  the  attached 
proposed  procedure  with  supervisory  personnel.  The  rationale  must  take 
into  account  the  type  of  equipment  contracted  for,  e.g.,  systems/subsys¬ 
tems/components,  etc. 

Tire  summary  should  be  prepared  on  each  contract  over  $500,000  at  the 
end  of  the  contract,  or  annually,  if  an  extended  period  is  involved  as  in  a 
system  contract.  The  summary  shou'  J  be  discussed  with  the  contractor 
and  a  copy  retained  in  the  plant  for  future  contracting/precontracting 
activities.  A  copy  will  also  be  provided  to  the  contracting  officer  and  to  the 
USAFCMD. 

The  individual  score  for  each  element/characterislic  should  be  assigned 
a  value  in  accordance  with  Table  1 : 


Table  1 


Qualir-  Valuation  Ratings 


Excellent 

91 

100 

Very  Good 

81 

90 

Good 

71 

80 

Marginal 

61 

70 

Submarginal 

0 

61 

79 


ATTACHMENT  1 

RECOMMENDED  CONTRACTOR  QUALITY 
PERFORMANCE  EVALUATION  SUMMARY 

Date  _ 

Contractor _ 

Contract  Number  _ 

Contract  Date _ 

Value  _ 

Number  of  Items  _ 

Evaluator  _ _ 


The  objective  of  the  evaluation  is  to  provide  a  continuing  assessment  of  the  firm’s 
quality  program.  The  data  below  provides  a  standard  rationale  for  selected  parts  of 
the  program.  The  assessment  is  limited  to  these  key  parts  to  minimize  and 
standardize  the  evaluation.  The  evaluator  must  assign  a  rating  from  table  1  which 
reflects  an  assessment  of  these  activities. 


Category  1 ,  Quality  of  Quality  Management  Systems 

Factor  =  .6 

Element 

Criteria  Ratinq 

Factor 

Effectiveness 

A.  Quality 

MIL-Q-9858, 

x  .2  = 

% 

Program 

Management 

Para.  3.2 

B.  Production 

MIL-Q-9858, 

x  .2  = 

% 

Processing 

and 

Para.  6.2 

Fabrication 

C.  Measuring 

MIL-Q-9858, 

x  .2  = 

% 

and  Testing 
Equipment 

Para.  4.2 

Total  Quality  Management  System  Score 

% 

Category  effectiveness  is  found  by  multiplying  the  rating  times  the  factor  (.2). 


Rating _ x  .2  = _ % 


Category  2,  Quality  of  Product  Conformance  Factor  =  .2 

The  objective  of  this  evaluation  is  to  provide  a  continuing  assessment  of  the 
conformance  of  products  to  design  requirements.  The  data  below  will  assist  the 
evaluator  in  making  what  is  primarily  a  subjective  assessment.  It  will  provide  one 
uniform  element  of  the  analysis  and  will  provide  a  continuing  evaluation  of  product 
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conformance.  The  evaluator  must  assign  a  score  from  Table  1  which  reflects  an 
overall  assessment  of  product  conformance  to  requirements. 

A.  Acceptance  Testing  Results 

(1 )  _ units  completed  acceptance  test  with  no  defects. 

(2)  _ units  required  retest  due  to  defects. 

B.  Manufacturing  Effectiveness 

(1 )  _ major  non-conformances  were  accepted. 

(2)  _ minor  non-conformances  were  accepted. 

Category  effectiveness  is  found  by  multiplying  the  rating  times  the  factor  (.2). 

Rating _ x  .2  = _ % 


Category  3,  Quality  of  Design  Factor  =  .2 

The  objective  of  the  evaluation  is  to  provide  a  continuing  assessment  of  the 
contractor’s  quality  of  design.  The  data  below  will  assist  the  evaluator  in  making 
what  is  primarily  a  subjective  assessment.  It  will  provide  one  uniform  element  of  the 
analysis  and  will  provide  a  continuing  evaluation  of  quality  of  design.  The  evaluator 
must  assign  a  score  from  Table  1  which  reflects  an  overall  assessment  of  design 
quality. 

Engineering  Changes 

(1)  _ Number  of  Class  1  changes  accepted. 

(2)  _ Number  of  Class  2  changes  accepted. 

(3)  _ Number  of  Class  1  changes  that  reduced  system  requirements. 

(4)  _ Number  of  Class  1  changes  submitted  requiring  further  changes 

to  parameters/characteristics  revised  in  earlier  approved  changes. 

Category  effectiveness  is  found  by  multiplying  the  rating  times  the  factor  (.2). 
Rating _ x  .2  = _ % 

OVERALL  QUALITY  PERFORMANCE  EVALUATION  SCORE 

Overall  contractor  score  is  found  by  summing  Category  1  score,  plus  Category  2 
score,  plus  Category  3  score. 

Overall  contractor  quality  rating  =  Category  1  score _ % 

Category  2  score _ % 

Category  3  score _ % 

Total  _ % 
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APPENDIX  B 


Quality  Award  Fee  Approach 


The  CPAF  contract  is  described  in  the  DAR  3-405.5  and  should  be 
reviewed  prior  to  development  of  an  Award  Fee  scheme  for  a  contract.  In 
the  event  that  a  decision  has  been  made  to  include  an  Award  Fee  feature, 
it  is  essential  that  the  Award  Fee  approach  be  defined  in  the  RFP  stage.  The 
RFP  should  include  an  explanation  of  the  Award  Fee  features  and  the  fee 
determining  procedures  that  will  be  employed.  The  RFP  must  also  identify 
the  criteria  that  will  be  used  to  evaluate  the  responsiveness  of  proposals  to 
the  Award  Fee  requirements.  The  Award  Fee  Determination  Plan  (AFDP)* 
should  be  included  as  a  separate  entity  with  the  RFP  package.  Considera¬ 
tions  should  be  given  to  allowing  prospective  contractors  to  provide  recom¬ 
mendations  for  possible  improvements  to  the  plan  as  a  draft  RFP. 

When  a  contract  is  issued  containing  an  Award  Fee  clause  for  quality,  it 
is  advantageous  to  provide  limited  references  to  the  Award  Fee  in  the 
contract  itself  and  to  retain  the  AFDP  as  a  separate  element.  This  will  allow 
the  flexibility  of  modifying  the  AFDP  without  modifying  the  basic  contract 
and  will  simplify  the  administrative  processes.  The  contract,  however, 
should  provide  some  coverage  regarding  the  Award  Fee  aspect  of  the 
contract  including  the  following: 

1.  The  Base  Fee  amount  {can  be  zero  but  should  be  substantial,  and 
may  be  influenced  by  other  Award  Fee  aspects  of  the  contract). 

2.  The  Maximum  Award  Fee  which  can  be  earned. 

3.  The  time  intervals  of  evaluation  and  pay  schedule  and  proportion 
which  can  be  paid  in  each  interval. 

4.  A  provision  that  fee  amounts  will  be  determined  unilaterally  by  the 
Fee  Determination  Official  (FDO). 

5.  A  provision  that  FDO  determinations  concerning  Award  Fee  earned 
is  binding  on  both  parties  and  not  subject  to  appeal  under  the 
dispute  clause. 

6.  A  provision  for  prompt  payment  of  fee  earned  after  each  determina¬ 
tion  without  a  need  for  contract  modification. 

7.  A  provision  allowing  equitable  adjustment  of  fee  in  the  event  of 
change  orders  or  other  contract  modifications. 

8.  A  provision  that  in  the  event  of  contract  cancellation  of  FDO 
determination  relative  to  the  amount  of  Award  Fee  available  will  be 
final  and  not  subject  to  the  disputes  clause. 

The  contract  should  also  identify  the  AFDP  by  date  and  title  and  include 
a  provision  that  the  FDO  has  a  unilateral  right  to  change  AFDP  coverage. 
However,  it  should  establish  that  the  contractor  will  receive  notice  of 
changes  by  some  specified  number  of  days  prior  to  the  evaluation  period  in 
which  the  changes  will  be  applied. 


•Abbreviations  shown  In  Kxhlblt  G.  page  105. 
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The  amount  of  fee  that  can  be  awarded  has  a  basic  limitation  in  DAR 
3-405. 6(c)(2).  However,  the  exact  amount  of  the  fee  must  be  assessed  in 
light  of  any  other  incentive  fees  that  may  be  a  part  of  the  contract.  Under 
ordinary  circumstances,  the  total  fee  allowed  for  a  contract  for  Research 
and  Development  is  limited  to  1 5  percent  of  the  estimated  cost  or  10  percent 
in  the  case  of  a  CPFF  contract. 

The  Fee  Determination  Official  and  board  members  must  be  chosen  with 
considerat  ion  of  their  availability  to  devote  adequate  time  to  the  plan  as  well 
as  their  position  in  the  government  hierarchy.  It  is  desirable  to  have 
i«  pi vsentatives  from  organizations  outside  of  the  plant  when  the  plan  is 
employed.  In  particular,  representation  from  the  using  command  is  highly 
desirable.  In  the  attached  format,  in  some  cases,  personnel  are  identified 
by  title.  This  is  only  for  illustrative  purposes.  The  evaluation  intervals  are 
also  examples.  Attachment  1  is  a  format  which  may  be  used  as  an  Award 
Fee  plan. 
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ATTACHMENT  1 

SAMPLE  AWARD  FEE  FORMAT  AND  APPROACH 
AWARD  FEE  DETERMINATION  PLAN  FOR 
Contract  No. _ with _ (tinn) 


Contents 
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1 .  GENERAL  TERMS  AND  CONDITIONS 

A.  This  plan  describes  the  method  of  administration  of  the  Award  Fee  provisions 

of  contract  no. _ __ _ ,  dated _ _ ,  with 

B.  The  term  of  the  contract  is  from _ through _ . 

1 .  The  estimated  cost  of  performing  the  contract  is  $ _ . 

2.  The  base  fee  is  $ _ . 

3.  The  maximum  Award  Fee  is  $ _ . 

4.  The  estimated  cost,  base  fee,  and  Award  Fee  are  subject  to  equitable 
adjustments  on  account  of  changes  or  other  contract  modifications. 

5.  The  Award  Fee  earned  and  payable  will  be  determined  periodically  by  the 
Fee  Determination  Official  in  accordance  with  this  plan. 

6.  Award  Fee  findings  and  decisions  are  not  subject  to  the  Disputes  clause  of 
the  contract. 

7.  The  FDO*  may  unilaterally  change  the  matters  in  this  plan,  as  covered  in 
part  5  and  not  otherwise  requiring  mutual  agreement  under  the  contract, 

providing  the  contractor  receives  notice  of  the  changes  at  least _ 

work  (or  calendar)  days  prior  to  the  beginning  of  the  evaluation  period  to 
which  th<  ranges  apply. 

C.  The  objective  of  the  Award  Fee  provisions  of  the  contract  is  to  afford  the 
contractor  an  opportunity  to  earn  increased  fee  commensurate  with  the 
achievements  of  optimum  performance  in  pursuit  of  contract  objectives  and 
goals. 

2.  MANAGEMENT  AND  ADMINISTRATION  OF  THE  AWARD  FEE  PROGRAM 

The  following  management  procedureisestablishedforadministeringthe  Award 
Fee  provisions  of  the  contract. 

A.  Fee  Determination  Official  (FDO) 

(1)  The  FDO  is _ Deputy  Director  SPO _ 

(2)  Primary  FDO  responsibilities  are: 

(a)  determining  the  Award  Fee  earned  and  payable  for  each  evalua¬ 
tion  period  as  addressed  in  Part  4. 

(b)  changing  the  matters  covered  in  this  plan  as  addressed  in  Part  5, 
as  appropriate. 

B.  Performance  Evaluation  Board  (PEB) 

(1 )  The  Chairman  of  the  PEB  is _ Director  of  Engineering  SPO 

The  following  are  voting  members: 

Using  Command  Representative  _ _ _ 


‘All  abbreviations  are  contained  in  Exhibit  G 
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Director  of  Contracting  SPO _ 

Director  of  Quality  Assurance  SPO _ 

Chief  of  Quality  Assurance  AFPRO  _ 

Other _ 

(2)  The  Chairman  may  recommend  the  appointment  of  non-voting  mem¬ 
bers  to  assist  the  Board  in  performing  its  functions. 

(3)  Primary  responsibilities  of  the  Board  are: 

(a)  Evaluate  the  contractor’s  performance  in  accordance  with  the 
performance  criteria  set  forth  in  the  contract  (and/or  in  this  plan). 
Evaluation  methods  will  include  but  not  be  limited  to: 

•  performance  monitor  reports 

•  other  reports  which  provide  useful  data 

(b)  Prepare  a  Performance  Evaluation  Board  Report  for  the  FDO, 
including  the  Board’s  findings  and  recommendations  for  each 
evaluation  period  as  addressed  in  Part  4. 

(c)  Periodically  review  the  evaluation  plan  and  its  implementation,  and 
recommend  necessary  changes  in  accordance  with  Part  5. 

C.  Performance  Monitors  (PMs) 

(1 )  The  PEB  Chairman  will  assign  a  PM  to  each  performance  area  to  be 
evaluated. 

(2)  Each  PM  will  be  responsible  for  complying  with  the  General  Instructions 
for  Performance  Monitors  in  Exhibit  B,  and  any  specific  instructions  of 
the  PEB  Chairman  including  the  following  primary  responsibilities: 

•  Monitoring,  evaluating,  and  assessing  contractor  performance  in 
assigned  activities. 

•  Periodically  preparing  a  Performance  Monitor  Report  (PMR)  for 
the  PEB. 

•  Recommending  appropriate  changes  in  this  plan  for  consideration 
in  accordance  with  Part  5. 

3.  EVALUATION  FACTORS 

Evaluation  requirements  are  attached  as  follows: 


A. 

Evaluation  Periods  and  Maximum 
Available  Award  Fee  for  Each 

Exhibit  A 

B. 

Performance  Factors  and  Evaluation 
Criteria  Exhibit  B 

C. 

Scoring  Table 

Exhibit  C 

D. 

Award  Fee  Conversion  Chart 

Exhibit  D 

The  percentages  described  in  Exhibit  B,  the  scoring  table  in  Exhibit  C,  and  the 
conversion  chart  in  Exhibit  D  are  provided  as  a  typical  methodology  for  a  general 
assessment  of  the  proportion  of  the  available  Award  Fee  that  has  been  earned. 
They  provide  only  sample  guidance  and  a  judgmental  evaluation  of  contractor 
performance  and  should  not  be  considered  to  be  the  results  of  a  rigorous  mathe¬ 
matical  determination. 

4.  AWARD  FEE  DETERMINATION  METHODOLOGY 

A  determination  of  the  Award  Fee  earned  for  each  evaluation  period  will  be  made 
promptly  by  the  FDO  after  the  end  of  the  period.  The  method  to  be  followed  in 
monitoring,  evaluating  and  assessing  contractor  performance  during  the  period,  as 
well  as  for  determining  the  Award  Fee  earned,  is  described  below.  Exhibit  E-1 
summarizes  the  principal  activities  and  recommended  schedules. 

A.  The  PEB  Chairman  will  assign  a  PM  for  each  performance  area  to  be 
evaluated  under  the  contract.  PMs  will  be  selected  on  the  basis  of  their 
expertise  relative  to  prescribed  areas  of  performance.  The  PEB  Chairman 
may  change  PM  assignments  at  any  time  without  advance  notice  to  the 
contractor  but  should  notify  the  contractor  as  promptly  as  possible. 

B.  The  PEB  Chairman  will  provide  to  each  PM  the  following  documents: 

(1)  A  copy  of  the  pertinent  elements  of  the  contract  and  subsequent 
modifications. 

(2)  A  copy  of  this  plan  along  with  any  changes  made. 

(3)  Sufficient  instructions  to  perform  the  PM  responsibilities. 

(4)  Any  unique  instructions  concerning  PM-assigned  performance  areas. 

C.  PMs  will  monitor,  evaluate  and  assess  contractor  performance  and  discuss 
the  results  with  appropriate  contractor  personnel,  in  accordance  with  the 
General  Instructions  for  Performance  Monitors  (Exhibit  E-2),  and  any  other 
specific  instructions  or  guidance  furnished  by  the  PEB  Chairman. 

D.  PMs  will  submit  monthly  PMRs  and,  if  required,  make  verbal  presentations 
to  the  PEB. 

E.  The  PEB  Chairman  will  consider  PMRs  and  request  and  obtain  performance 
information  from  other  units  or  personnel  that  may  assist  in  determining 
contractor  performance.  He  will  discuss  reports  and  information  with  PMRs 
and  other  personnel  as  he  deems  necessary. 

F.  Promptly  after  the  end  of  each  month,  except  the  final  month  of  the  evaluation 
period,  the  PEB  Chairman  will  meet  with  the  contractor  and  discuss  overall 
performance  during  the  period.  If  requested  by  the  PEB  Chairman,  PMs  and 
other  personnel  involved  in  performance  evaluations  will  attend  the  meeting 
and  participate  in  discussions. 

G.  Promptly  after  the  end  of  each  evaluation  period,  the  PEB  will  meet  and 
consider  all  performance  information  obtained,  summarize  preliminary  find¬ 
ings,  and  determine  recommendations  for  coverage  in  the  PEBR. 
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H.  Subsequently,  the  PEB  Chairman  will  meet  with  the  contractor  Award  Fee 
representative  and  discuss  preliminary  findings  and  recommendations.  As 
requested  by  the  PEB  Chairman,  PMs  and  other  personnel  involved  in 
performance  evaluation  will  attend  the  meetings  and  participate  in  discus¬ 
sions.  At  this  meeting,  the  contractor  representative  will  be  given  an  oppor¬ 
tunity  to  submit  matters  in  the  firm’s  behaif,  including  an  assessment  of 
performance  during  the  evaluation  period. 

I.  After  meeting  with  contractor  personnel,  the  PEB  will  consider  matters 
presented  by  contractor  personnel  and  establish  Board  findings  and  recom¬ 
mendations  for  the  PEBR. 

J.  The  PEB  Chairman  will  prepare  the  report  for  the  period  and  submit  it  to  the 
FDO  for  use  in  determining  the  Award  Fee  earned.  The  report  will  include 
a  recommended  range  of  Award  Fee  with  supporting  documentation.  When 
submitting  the  report,  the  Chairman  will  inform  the  FDO  whether  or  not  the 
contractor’s  representative  desires  to  present  any  matters  to  the  FDO  before 
the  Award  Fee  determination  is  made. 

K.  The  FDO  will  review  the  PEBR  and  discuss  it  with  the  PEB  Chairman  and 
other  personnel,  as  appropriate.  If  requested  by  the  contractor,  or  if  the  FDO 
considers  it  appropriate,  the  FDO  will  meet  with  the  contractor  for  discus¬ 
sions.  If  requested  by  the  FDO,  the  PEB  Chairman  and  any  other  personnel 
involved  in  performance  evaluation  may  be  required  to  attend  the  meeting 
with  the  contractor. 

L.  The  FDO  will  determine  the  amount  of  Award  Fee  earned  during  the  period. 
The  amount  determined  will  include  the  results  of  judgmental  assessments 
as  well  as  utilizing  the  analytical  procedures  herein.  The  FDO's  determina¬ 
tion  of  the  amount  of  Award  Fee  earned  and  the  basis  of  this  determination 
will  be  stated  in  the  Award  Fee  Determination  Report  (AFDR).  The  report 
will  be  signed  by  the  FDO  and  provided  to  the  contractor  as  justification  for 
payment  of  the  Award  Fee  granted. 


5.  CHANGES 

A.  Unilateral  Changes 

The  FDO  may  unilaterally  change  any  matters  covered  in  this  plan  not 
otherwise  requiring  mutual  agreement  under  the  contract,  prior  to  the  begin¬ 
ning  of  any  evaluation  period  by  timely  notice  to  the  contractor  in  writing. 
The  changes  will  be  made  without  formal  modification  of  the  contract. 

B.  Method  For  Making  Changes 

The  method  to  be  followed  for  changing  plan  coverage  follows.  Exhibit  F 
summarizes  the  principal  actions  and  schedules  involved. 

(1 )  Any  person  involved  in  the  administration  of  the  Award  Fee  provisions 
of  the  contract  may  recommend  any  changes  in  the  plan  which  will  result 
in  improved  performance  or  improve  the  Award  Fee  determination 


91 


process  by  providing  the  recommended  changes  to  the  PEB  for  his 
evaluation. 

(2)  Any  proposed  changes  will  be  coordinated  with  the  contractor  by  the 
PEB  Chairman. 

(3)  Prior  to  the  end  of  each  evaluation  period,  the  PEB  will  submit  changes 
applicable  to  the  next  evaluation  period  for  approval  by  the  FDO  with 
appropriate  comments  and  justification,  or  inform  the  FDO  that  no 
changes  are  recommended  for  the  next  period. 

(4)  ♦  work  days  before  the  beginning  of  each  evaluation  period,  the 
FDO  will  notify  the  contractor  in  writing  if  any  changes  will  be  made.  If 
any  such  notification  is  not  provided  to  the  contractor  within  the  agreed- 
to  number  of  work  days  before  the  beginning  of  the  next  period,  existing 
plan  coverage  will  continue  in  effect  for  the  next  evaluation  period. 


Number  of  days  are  to  be  agreed  upon  by  both  parties  during  contract  negotiations. 
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EXHIBIT  A  TO  AFDP  FOR  CONTRACT  # _ 

Contract  No. _ with _ 

EVALUATION  PERIODS  AND  MAXIMUM 
AVAILABLE  AWARD  FEE  FOR  EACH 

Evaluation  Period 


Max.  Available 

No.  Duration*  Ending  Award  Fee 

1  3  months 

2  6  months 

3  6  months 

4  6  months 

5  6  month  intervals 
to  end  of  contract 

Maximum  Fee  Each  Evaluation  Period  =  Total  Fee 

Number  of  evaluation  periods 


‘Periods  and  equal  fee  distributions  here  are  examples  only.  If  the  fee  is  varied  for  different  periods,  the 
evaluation  equation  must  be  modified 
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EXHIBIT  B  TO  AFDP  FOR  CONTRACT  # _ 

PERFORMANCE  FACTORS  AND  EVALUATION  CRITERIA 


The  performance  areas  to  be  evaluated  are  identified  below. 
The  evaluation  criteria  for  each  area  are  attached,  as  indicated. 


Factor 

Factor 

See 

Area  No. 

Identification 

Weiaht 

Exhibit 

1 

Quality  System 
Management 

.25 

B-1 

2 

Production  of 
Nonconforming 

Material 

.25 

B-2 

3 

Major  Assembly 
Inspection  Findings 

.25 

B-3 

4 

Completed  End 

.25 

B-4 

Item  Quality  Assessment 
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EXHIBIT  B-1  TO  AFDP  FOR  CONTRACT# _ 

EVALUATION  CRITERIA  FOR  PERFORMANCE  FACTOR  NO.  1 

QUALITY  SYSTEM  MANAGEMENT  Factor  Weight  .25 

Description  of  Element: 

Para.  3  MIL-Q-9858A 

The  objective  of  the  element  is  to  encourage  corporate  wide  involvement  in  the 
quality  management  system. 

Sub-Elements  to  Consider: 

Para.  3.1  Organization,  3.4  Records,  3.5  Corrective  Action,  3.6  Costs  related  to 
quality. 

Evaluation  Criteria: 

Evaluation  will  be  per  MIL  Handbook  H50,  Para.  3.1,  3.4,  3.5,  and  3.6  except 
that  complete  cost  data  should  be  available  for  prevention,  appraisal,  and  failure 
(both  internal  and  external  costs). 

Basis  or  Standard  for  Measuring  Performance: 

This  is  a  subjective  appraisal  of  a  firm’s  performance  as  evaluated  by  the  PM. 
It  is  to  be  used  in  consonance  with  the  narrative  instruction  in  the  rating  table 
(Exhibit  C). 

Sub-Element  Weights: 


3.1  -  .3 

Para.  3.1 

Score 

x  .3 

3.4  -  .2 

Para.  3.4 

Score 

x  .2 

3.5  -  .3 

Para.  3.5 

Score 

x  .3 

3.6  -  .3 

Para.  3.6 

Score 

x  .2 

Total 
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EXHIBIT  B-2  TO  ADFP  FOR  CONTRACT# 


EVALUATION  CRITERIA  FOR  PERFORMANCE  FACTOR  NO.  2 

NON-CONFORMING  MATERIAL  Factor  Weight  .25 

Description  of  Element: 

The  objective  of  this  factor  is  to  develop  a  manufacturing  process  in  which  the 
production  of  defective  material  is  the  consequence  of  only  random  events. 

Sub-Element  to  Consider: 

(1)  Purchased  material,  (2)  machine/process  functions,  (3)  assembly,  (4) 
productivity 

Evaluation  Criteria: 

1 .  Is  there  an  approved  overall  company  cost  factor  for  manufacturing  losses 
resulting  from  costs  associated  with  the  procurement  of  defective  material? 

2.  Are  man/machine  process  capability  studies  performed  to  analyze  produc¬ 
tion  problems? 

3.  Are  process  control  records  maintained  at  assembly  stations  that  identify 
recurring  defects? 

4.  Is  there  a  positive  program  to  publicize  improvements  in  productivity 
resulting  from  quality  improvements? 

Basis  or  Standard  for  Measuring  Performance: 

This  is  a  subjective  appraisal  of  the  firm’s  performance  as  evaluated  by  the  PM. 
It  is  to  be  used  in  consonance  with  the  narrative  instructions  in  the  rating  table 
(Exhibit  C). 

Sub-Element  Weights: 


1.  - 

.25 

Score 

_  X 

.25 

v.  _ 

.25 

Score 

_  X 

.25 

3.  - 

.25 

Score 

_  X 

.25 

4.- 

.25 

Score _ 

X 

.25 

Total 
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EXHIBIT  B-3  TO  AFDP  FOR  CONTRACT  # 


EVALUATION  CRITERIA  FOR  PERFORMANCE  FACTOR  NO.  3 
MAJOR  ASSEMBLY  INSPECTION  Factor  Weight  .25 

Description  of  Element: 

The  objective  of  this  element  is  to  assure  the  development  of  a  system  to  analyze 
and  eliminate  non-random  manufacturing/assembly/test  problems  that  con¬ 
tribute  to  end  item  testing  problems,  delays  and  costs. 

Sub-Elements  to  Consider: 

All  areas  where  subsystems  or  major  components  are  given  a  final  checkout 
prior  to  assembly  into  the  final  item/system  configuration.  In  the  interests  of 
focusing  attention  sub-elements  should  not  exceed  1 0. 

Evaluation  Criteria: 

The  criteria  should  concentrate  on: 

1 .  identification  of  significant  problems 

2.  analysis  as  to  their  cause 

3.  promptness  and  effectiveness  of  corrective  action  taken 
Basis  for  Measuring  Performance: 

This  is  a  subjective  appraisal  of  the  firm’s  performance  as  evaluated  by  the  PM. 
It  is  to  be  used  in  consonance  with  the  narrative  instructions  in  the  rating  table 
(Exhibit  C). 

Sub-Element  Weights: 

The  evaluation  must  determine  how  many  sub-elements,  (subsystem  checkout 
stations)  there  are.  It  is  recommended  that  an  equal  weight  be  assigned  to  each 
and  that  a  table  and  scoring  procedure  similar  to  that  in  Exhibit  B-1  be  prepared, 
e.g.  -  * 


Sub-Element  1  -  .33 

Score 

x  .33  = 

Sub-Element  2  -  .33 

Score 

x  .33  = 

Sub-Element  3  -  .33 

Score 

x  .33  = 

Total 

'The  number  of  elements  is  immaterial  but  the  total  score  cannot  exceed  1 .0. 


EXHIBIT  B-4  TO  AFDP  FOR  CONTRACT# _ 

EVALUATION  CRITERIA  FOR  PERFORMANCE  FACTOR  NO.  4 

COMPLETED  END  ITEM  QUALITY  ASSESSMENT  Factor  Weight  .25 

Description  of  Element: 

The  objective  of  this  element  is  to  generate  a  measure  of  product  quality  in  terms 
of  conformance  with  design  requirement.  It  provides  product  conformance 
information  both  in  the  manufacturer’s  facility  and  after  delivery. 

Sub-Elements  to  Consider: 

1 .  Number  of  defects  found  by  contractor  in  final  end  item  inspection/test  on 
each  end  item. 

2.  Number  of  minor  waivers  granted  on  each  end  item. 

3.  Number  of  major  waivers  granted  on  each  end  item. 

4.  Number  of  defects  found  by  user  if  inspection  is  performed. 

5.  Number  of  user  generated  deficiency  reports. 

Evaluation  Criteria: 

In-plant  final  inspection 
User  inspection  activities 
User  routine  deficiency  reports 

Basis  or  Standard  for  Measuring  Performance: 

This  is  a  subjective  appraisal  of  the  firm's  performance  as  evaluated  by  the  PM. 
It  is  to  be  used  in  consonance  with  the  narrative  instructions  in  the  rating  table 
(Exhibit  C). 

Sub-EiementWeights: 


1  -  .2 

Sub-Element  1 

Score 

x  .2  = 

2  -  .1 

Sub-Element  2 

Score 

x  .1  = 

3 -.3 

Sub-Element  3 

Score 

x  .3  = 

4-  .3 

Sub-Element  4 

Score 

x  .3  = 

5  —  .1 

Sub-Element  5 

Score 

x  .1  = 

Total 
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EXHIBIT  C  TO  AFDP  FOR  CONTRACT  # 

RATING  TABLE 


Proportion  of  Maximum 


Narrative  Grade 

Description  Achievable  Score  Available 

Excellent 

Performance  far  exceeds  that 
normally  achieved  for  a  similar 
task.  Few  errors  are  made. 
Management  and  employee 
controls  outstanding. 

90-100 

Good 

Performance  exceeds  that  nor¬ 
mally  achieved.  Above 
average  achievement  of  all 
areas  of  element. 

60-90 

Satisfactory 

Performance  meet  minimum 
standards.  No  exceptional 
management  emphasis  on 
quality,  problems  encountered, 
routinely  resolved. 

30-50 

Marginal 

Performance  less  than  ex¬ 
pected.  Continuing  extraordi¬ 
nary  efforts  required  to  assure 
material  is  meeting  require¬ 
ments. 

10-20 

Unsatisfactory 

Performance  not  acceptable. 
Material  cannot  be  shipped 
without  government  inspection. 
Quality  management  system 
not  accepted. 

0 
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EXHIBIT  D  TO  AFDP  FOR  CONTRACT  # _ 

The  following  table  converts  achieved  Award  Fee  scores  into  fee  earned. 

Performance  Scoies: 

Factor  No.  1 ,  Score _ x  .25  =  Available  %  of  Award  Fee  __ 

Factor  No.  2,  Score _ x  .25  =  Available  %  of  Award  Fee _ _ 

Factor  No.  3,  Score _ x  .25  =  Available  %  of  Award  Fee  __ 

Factor  No.  4,  Score _ x  .25  =  Available  %  of  Award  Fee _ 

Total  %  available  Award  Fee  earned  _ _ 

Total  %  of  Award  Fee  earned _ x  available  fee__ _ _= _ fee  earned. 


(00 


EXHIBIT  E-1  TO  AFDP  FOR  CONTRACT  # _ 

The  following  is  a  summary  of  the  principal  actions  involved  in  determining  the 
Award  Fee  for  each  evaluation  period. 


Action 

Schedule 
{Work  Days) 

PED  Chairman  appoints  PMs, 
provides  necessary  guidance, 
and  informs  contractor 

as  required 

PMs  assess  performance  and 
discuss  results  with  contractor 

throughout  the  period 

PMs  submit  PMRs  to  PEB 

last  day  of  each  month 

PEB  obtains  performance 
information  from  other 
procuring  activity  personnel 

as  necessary 

PEB  considers  PMRs  and 
any  other  available 
performance  information 

continuing 

PEB  Chairman  discusses 

days  after  end 

overall  performance  with 

of  each  month,  except 

contractor  during  period 

last  month  in  period 

PEB  meets  and  prepares 
preliminary  findings  and 

days  after  end 

position  for  PEBR 

of  period 

PEB  Chairman  meets  with 
contractor  and  discusses 

days  after  end 

preliminary  findings  and  position 

of  period 

PEB  establishes  findings 

days  after  end 

and  recommendations  for  PEBR 

of  period 

PEB  Chairman  submits 

days  after  end 

PEBR  to  FDO 

of  period 

FDO  reviews  PEBR  and 
discusses  it  with 

days  after  end 

PEB,  as  appropriate 

of  period 

FDO  sends  AFDR  to 
contractor  and 

days  after  end 

contracting  office 

of  period 

The  PEBD  will  establish  appropriate  lists  of  subsidiary  actions  and  schedules  to 
meet  the  above  schedules,  as  necessary. 
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EXHIBIT  E-2  TO  AFDP  FOR  CONTRACT  # 


PERFORMANCE  MONITORS  INSTRUCTIONS 


1 .  Monitoring  and  Assessing  Performance 

a.  PMs  will  prepare  outlines  of  their  assessment  plans  as  a  part  of  the  PEB 
documentation.  The  plan  snuuld  be  discussed  with  appropriate  contractor 
personnel  to  assure  understanding  of  the  evaluation  and  assessment 
methodology. 

b.  PMs  will  plan  and  carry  out  both  scheduled  and  unannounced  assessment 
visits;  however,  before  each  scheduled  visit,  the  PM  should  contact  ap¬ 
propriate  contractor  personnel  who  may  accompany  the  PM,  if  desired. 

c.  PMs  will  conduct  all  assessments  in  an  open,  objective  and  cooperative  spirit 
so  that  a  fair  and  valid  evaluation  is  achieved.  This  will  enable  contractor 
personnel  to  plan  improvements  in  performance.  Positive  performance 
accomplishments  must  be  recognized  as  well  as  inadequate  results. 

d.  After  assessments,  the  PM  will  discuss  the  results  with  appropriate  contrac¬ 
tor  personnel,  noting  any  observed  deficiencies  and/or  accompanying 
recommendations.  Areas  of  poor  performance  will  be  discussed  to  afford 
the  contractor  an  opportunity  to  clarify  possible  misunderstandings  and  to 
resolve  deficiencies. 

e.  PM  contacts  with  contractor  peisonnel  are  to  be  accomplished  within  the 
context  of  official  contractual  relationships.  Any  activity  or  association  which 
might  cause,  or  give  the  appearance  of  causing,  a  conflict  of  interest  must 
be  avoided. 

f.  PM  discussions  with  contractor  personnel  are  not  to  be  used  to  instruct  or 
direct  contractor  pe  >  '.-.one  he  performance  of  a  contract. 

2  Documenting  Evaluations/Assessments 

Evaiuatiuub  and  assessments  conducted,  results  obtained,  and  discussions  with 
contractor  personnel  will  be  documented  immediately  after  each  such  actions 
by  preparing  a  brief  summary  of  observations  and  discussions  with  contractor 
personnel. 

3.  Evaluation/Assessment  Reports 

PMs  will  prepare  a  normal  PMR  in  accordance  with  the  following  instructions 
and  submit  it  to  the  PEB,  or  others,  if  appropriate,  at  the  end  of  each  month. 
Information  essential  to  the  PMR  element  involved  should  be  identified  and  a 
format  specified  if  deemed  advisable. 
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4.  Verbal  Reports 

PMs  will  be  prepared  to  make  verbal  reports  as  required  by  the  PEB  Chairman. 

5.  Performance  Monitor  Report 

The  PMR  will  contain,  as  a  minimum,  a  record  of  each  meeting  or  discussion  in 
connection  with  the  Award  Fee  reviews,  including  the  names  of  contractor 
personnel,  date  of  meeting(s),  and  a  brief  summary  of  discussions.  Any  differ¬ 
ences  of  opinion  with  contractor  personnel  must  be  included  with  a  full  and 
complete  discussion  of  the  issues. 
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EXHIBIT  F  TO  AFDP  FOR  CONTRACT  # _ 

CONTINGENCY  PLAN  FOR  CHANGING  PLAN  COVERAGE 


The  following  is  a  summary  of  the  principal 
coverage. 


Action 

1 .  PEB  drafts  proposed  changes 

2.  PEB  coordinates  proposed 
changes  with  contractor 

3.  PEB  submits  recommended 
changes  to  FDO 

4.  FDO  notifies  contractor 
of  changes  or  that  there 
are  no  changes 


actions  involved  in  changing  plan 

Schedule 
(Work  Days) 

as  necessary 


as  necessary 
_ days 

prior  to  end  of  each  period 

_ _days 

before  start  of 
applicable  period 
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EXHIBIT  G  -  LIST  OF  TERMS  AND  ABBREVIATIONS 

The  following  is  an  alphabetical  list  of  all  acronyms  used  throughout  Appendix 
VI-3: 


AF 

Award  Fee 

AFDP 

Award  Fee  Determination  Plan 

AFDR 

Award  Fee  Determination  Report 

CAFR 

Contractor  Award  Fee  Representative 

CPAF 

Cost-Plus  Award  Fee 

FDO 

Fee  Determination  Official 

FM 

Function  Monitor 

FMR 

Function  Monitor  Report 

FPI 

Fixed-Price  Incentive 

ICC 

Incentive  Contracting  Committee 

PEB 

Performance  Evaluation  Board 

PEBC 

Performance  Evaluation  Board  Chairman 

PEBR 

Performance  Evaluation  Board  Report 

PEC 

Performance  Evaluation  Committee 

PECR 

Performance  Evaluation  Committee  Report 

PM 

Performance  Monitor 

PMR 

Performance  Monitor  Report 

RFP 

Request  For  Proposal 
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APPENDIX  C 
LETTERS 


DEPARTMENT  OF  THE  AIR  FORCE 

HEADQUARTERS  AIR  FORCE  SYSTEMS  COMMAND 
ANDREWS  AIR  FORCE  BASE  DC  20334-3000 


BtPLrTO  t  Tr- 

ATTN  Or  PL  J  19.'' 

suucct  Research  Questionnaire 


TO  Quality  Improvement  Survey  Participant 

1 .  As  the  Deputy  Chief  of  Staff  for  Product  Assurance  and  Acquisition  Logistics 
at  HQ  AFSC,  I  want  to  ensure  that  we  are  doing  what  we  should  to  acquire  the 
highest  quality  weapon  systems  in  the  most  effective  manner  possible.  That  is 
why  the  AFSC  Research  Fellow  at  the  Air  Command  and  Staff  College  has  developed 
this  survey.  You  have  been  selected  to  receive  this  questionnaire  because  of 
your  position  and  experience  in  the  acquisition  community.  I  encourage  you  to 
take  the  time  to  fully  complete  the  survey  based  on  your  present  knowledge  and 
abilities. 


2.  The  purpose  of  this  research  is  to  see  what  can  and  should  be  done  to 
improve  the  level  of  quality  in  our  weapon  systems  by  assessing  the  practices  we 
currently  use.  We  will  be  surveying  program  directors,  program  managers,  pro¬ 
ject  managers  and  quality  assurance  managers  in  AFSC’ 3  product  divisions.  Your 
response  will  remain  anonymous.  Please  do  not  put  your  name  or  social  security 
number  on  the  answer  sheet.  When  you  have  completed  the  questionnaire,  return  it 
in  the  enclosed  postage  paid  envelope  with  any  comments  you  wi3h  to  provide. 
Please  respond  no  later  than  29  Feb  88. 


3.  I  sincerely  appreciate  your  participation  in  this  important  research  effort. 
If  you  would  like  any  feedback  on  the  results  of  this  survey,  you  may  contact 
my  project  officer,  Maj  John  Campbell,  AUTOVON  875-2207. 


— 


DAVID  J.  TEAL,  Maj  Gen,  USAF 
DCS/ Product  Assurance  and 
Acquisition  Logistics 


UNITED  STATES  AIR  FORCE 


SEPTEMBER  18,1947 
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DEPARTMENT  OF  THE  AIR  FORCE 
AIH  UNIVERSITY 

CENTER  FOR  AEROSPACE  DOCTRINE.  RESEARCH,  AND  EDUCATION 
MAXWELL  AIR  FORCE  BASE.  AL  3S1 12-5532 


fl6PLT  TO 
ATTN  OF 


RIC 


3  MAR  Yil 


Quality  improvement  Research  Questionnaire 


Program  Director 

1.  A  few  weeks  ago  I  sent  you  letter  from  Major  General  David 
Teal,  DCS/Product  Assurance  and  Acquisition  Logistics,  HQ/AFSC. 
His  letter  requested  you  to  fill  out  a  questionnaire  on  quality 
improvement  research  that  I  am  doing  for  HQ  AFSC.  If  you  have 
already  returned  the  questionnaire,  please  disregard  this  letter. 
However,  if  you  have  not  done  so,  please  let  me  encourage  you  to 
do  so  as  soon  as  possible,  it  is  important  for  the  validity  of 
this  research  that  I  receive  your  response  no  later  than  18  March 
1988.  If  you  have  lost  the  questionnaire  or  did  not  receive  one 
please  call  me  at  AUTOVON  875-2207  and  I  will  immediately  mail 
you  one. 


2.  I  realize  that  you  are  very  busy  and  that  I  am  imposing  an 
additional  burden  upon  you,  but  I  think  your  comments  are 
important  and  with  them  we  might  improve  how  we  manage  quality  in 
tw  future,  jp-h^n.k  ifoyr  in  advance  for  your  time  and  effort. 


JOHN  A.  CAMPBEtL,  Major,  USAF 
HQ  AFSC  Research  Fellow 


Strength  Through  Knowledge 
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DEPARTMENT  OF  THE  AIR  FORCE 
AIR  UNIVERSITY 

CENTER  FOR  AEROSPACE  DOCTRINE,  RESEARCH,  AND  EDUCATION 
MAXWELL  AIR  FORCE  BASE.  AL  361 1 2-S532 


OEPL-  '0 

AfTN  OP 


RIC 


Quality  Improvement  Research  Questionnaire 


3  MAR  '=o8 


AFSC  Quality  Assurance  Managers 

1.  A  few  weeks  ago  I  sent  you  letter  from  Major  General  David 
Teal,  DCS/Product  Assurance  and  Acquisition  Logistics,  HQ/AFSC. 
His  letter  requested  you  to  fill  out  a  questionnaire  on  quality 
improvement  research  that  I  am  doing  for  HQ  AFSC.  If  you  have 
already  returned  the  questionnaire,  please  disregard  this  letter. 
However,  if  you  have  not  done  so,  please  let  me  encourage  you  to 
do  so  as  soon  as  possible.  It  is  important  for  the  validity  of 
this  research  that  I  receive  your  response  no  later  than  18  March 
1988.  If  you  have  lost  the  questionnaire  or  did  not  receive  one 
please  call  me  at  AUTOVON  875-2207  and  I  will  immediately  mail 
you  one. 


2.  I  realize  that  you  are  very  busy  and  that  I  am  imposing  an 
additional  burden  upon  you,  but  I  think  your  comments  are 
important  and  with  them  we  might  improve  how  we  manage  quality  in 
the  future.  TJumk  you. in  advance  for  your  time  and  effort. 


)HN  A.  CAMPBELL,  Major,  USAF 
''HQ  AFSC  Research  Fellow 


Strength  Througn  * -owiedge 
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APPENDIX  D 

PROGRAM  DIRECTOR  AND  MANAGER 
QUESTIONNAIRE 


QEEST/ONNA/EE 


MAJ  JOHN  A.  CAMPBELL 

Airpower  Research  Institute 

Maxwell  Air  Force  Base  AL 
36112 


Survey  Control  Number  USAF  SCN  87-  130B 


INSTRUCTION'S 


1.  Thank  you  for  taking  the  time  to  fill  out  this  questionnaire. 
M  o  r  ■  -  f  the  answers  will  require  you  to  respond  on  a  five  point 

s'..  Since  there  are  no  "right"  or  "wrong"  answers  and  many 

ns  ask  for  your  opinion,  please  mark  the  answer  that  most 
represents  your  position. 

... .  :  can  use  this  booklet  to  mark  your  answers  and  then 

t  ran  -t  hem  to  the  answer  sheet  provided.  Please  use  a  black, 
soft  Lead  pencil  (No.  2)  on  the  answer  sheet.  When  you  are  done 
please  return  the  answer  sheet  in  the  enclosed  addressed 
envelope.  If  you  would  like  to  provide  additional  information, 
either  enclose  an  additional  sheet  of  paper  with  your  comments  or 
make  your  comments  on  the  ques t iona i r e . 

3.  Please  try  to  answer  each  question,  however  if  no  answer 
applies  or  you  do  not  have  the  knowledge  to  answer  it,  then  skip 

it  and  go  to  the  next  question. 

4.  Your  name  or  organization  is  not  needed  for  this  study. 
However,  if  you  would  like  to  receive  a  brief  summary  of  the 

of  this  study .  ol  ease  provide  your  name  and  address.  You 
your  or  g  ini  o:  11  t  in  any  way  be  identified  in  this 

study 

o.  ’  ,i-ir.  •  v u  for  your  time  and  cooperation. 


J  0  :•  N  A.  CAMPBELL,  Maj,  USAF 
tp  A ESC  Research  Fellow 
Airpower  Research  Institute 
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WORK  FORCE  PROFILE 


1.  What  is  your  military/civilian 
equivalent  rank? 


GS/M  16 
MIL  B/G 
a 


15  14 

Col  LTC 

b  c 


13  <  12 

MAJ  <  CPT 

d  e 


2. 

How  many  years  have  you  worked  in 

>  10 

9-8 

7-6 

5-4 

£  3 

a  program  management  function? 

a 

b 

c 

d 

e 

3. 

How  many  different  systems/items 

>  5 

4 

3 

2 

1 

have  you  managed? 

a 

b 

c 

d 

e 

4. 

How  many  years  have  you  worked  on  this 

>  5 

4 

3 

2 

1 

program  (your  current  one)? 

a 

b 

c 

d 

e 

5. 

Are  you  the  program  director/program 

manager  versus  a  project  manager  on 

YES 

NO 

a  system/subsystem? 

a 

b 

6  . 

Your  system/item  is  primarily  in  which 

Concep 

Val 

PSD 

Prod 

Deploy 

acquisition  phase? 

a 

b 

c 

d 

e 

7  . 

How  many  full-time  quality  personnel 

>_  4 

3 

2 

1 

0 

are  assigned  to  your  organization? 

a 

b 

c 

d 

e 

8. 

How  much  of  your  day  (hours)  is  spent 

>  5 

4-3 

2 

1 

0 

on  quality  issues  (failures/corrections) 

?  a 

b 

c 

d 

e 

9. 

How  many  weeks  of  AF  training  in 

>  10 

9-8 

7-6 

5-4 

<  3 

program  management  have  you  had? 

a 

b 

c 

d 

e 

10. 

How  many  hours  of  tr aining/educat i ~n 

have  you  had  specifically  addressing  how 

>  40 

39-30 

29-20 

19-10 

<  9 

to  improve  quality  hardware/services? 

a 

b 

c 

d 

e 

11 . 

How  many  courses  to  improve  product 

performance  and  durability  have  you 

>  4 

3 

2 

1 

0 

attended  within  the  last  2-3  years? 

a 

b 

c 

d 

e 

SYSTEM  PROGRAM  QUALITY 

Use  the  scale  at  right  to  answer 

questions  12-15. 

12.  How  often  do  you  interface  directly 
with  the  SPO  personnel  who  track  quality 
performance? 

13.  How  often  do  you  personally  review 
user  complaint  data/trends  on  your 
system/item? 


Daily  weekly  Monthly  Qtrly  never 


a 


b 


c  d  e 


a 


b 


c  d  e 


117 


Daily  Meekly  Monthly  Qtrly  Never 

14.  How  often  do  you  personally  review 
your  contractor's  rework,  repair, 

retest  scrap,  etc.,  levels?  a  b  c  d  e 

15.  How  often  do  you  personally  review 
your  contractor's  "cost  of  quality" 

(failure  costs  +  routine  inspec¬ 
tion  +  prevention  costs)  data/trends?  a  b  c  d  e 


16.  How  many  quality  of  conformance  problems 
have  had  significant  program  impacts 

during  the  past  year  (i.e.,  schedule,  >10  5310 

cost,  accident,  publicity,  etc.)?  a  b  c  d  e 

17.  Jls  a  percentage  of  total  contract  value, 

what  would  you  estimate  the  "cost  25%  24-20%  19-15%  14-10%  £  9% 

of  quality"  is  for  your  program?  a  b  c  d  e 


Use  the  scale  at  right  to  answer  questions 
18-20.  These  questions  are  asking  you  to 
rate  the  breakdown  of  the  cost  of  quality. 

If  you  don't  know,  leave  it  blank.  >_  90%  89-70%  69-50%  49-30%  £  29% 

18.  What  percentage  does  failure  cost 

contribute  to  your  cost  of  quality?  a  b  c  d  e 

19.  What  percentage  does  routine  factory 
inspection  contribute  to  your  cost 

of  quality?  a  b  c  d  e 

20.  What  percentage  does  prevention  measures 

contribute  to  your  cost  of  quality?  a  b  c  d  e 


21.  What  degree  of  confidence  do  you  have 

in  the  accuracy  for  your  cost  of  j>  95%  94-75%  74-50%  49-25%  <_  25% 

quality  estimate?  a  b  c  d  —  e 


The  next  qrcup  of  questions  will  help  us  determine  from  your  perspective 
how  best  to  improve  quality  assurance  management.  Each  question  is 
asking  to  what  extent  you  think  the  statement  applies  to  you/your 
organization.  Therefore,  each  question  should  be  preceded  by  TO  WHAT 

EXTENT.  If  you  don't  know  the  answer,  leave  it  blank. 


Use  the  scale  at  right  to  answer 
questions  22-70. 

22.  Does  your  user  influence  contractor 


Not  Very  To 

at  all  Little  Soae 

extent 


To  a  To  a 
Great  Very  Grt 
Extent  Extent 


quality  of  hardware/services? 


a  b  c  d  e 
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TO  WHAT  EXTENT 


■ot  T«ry  To 

at  all  Little  Soae 

latent 


To  a  To  a 
Creat  Very  Crt 
latent  latent 


23  . 

Have  delivery  schedules  taken  priority 
over  quality  decisions? 

a 

b 

c 

d 

e 

24  . 

Do  engineering  errors  lead  to  problems 
in  managing  your  system/item? 

a 

b 

c 

d 

e 

25. 

Do  manufacturing  errors  lead  to  problems 
in  managing  your  system/item? 

a 

b 

c 

d 

e 

26  . 

Do  quality  assurance  personnel  help  you 
achieve  the  expected  level  of  quality? 

a 

b 

c 

d 

e 

27  . 

Are  you  able  to  influence  the  contractor 
to  provide  quality  products  using 
contract  requirements? 

a 

b 

c 

d 

e 

28. 

Are  you  able  to  influence  the  contractor 
to  provide  quality  products  using  good 
business  practices? 

a 

b 

c 

d 

e 

29. 

would  a  Firm  Pixed  Price  type 
contract  influence  your  contractor 
to  provide  a  quality  product? 

a 

b 

c 

d 

e 

38. 

Would  an  award  fee  influence  your 
contractor  to  provide  a  quality  product? 

a 

b 

c 

d 

e 

31. 

Would  a  warranty  influence  your  contractor 
to  provide  a  quality  product? 

a 

D 

c 

a 

e 

CO 

ro 

Has  your  contractor  used  employee 
motivation  techniques  to  improve 
quality  of  products? 

a 

b 

c 

d 

e 

33. 

Has  your  contractor's  top  management's 
attention  influenced  the  quality 
of  the  product  for  your  user? 

a 

b 

c 

d 

e 

34. 

Do  your  quality  personnel  focus  on  defect 
prevention  rather  than  detection? 

a 

b 

c 

d 

e 

35. 

Do  you  think  the  number  of  quality 
problems  in  the  last  year  were  normal 
for  a  program  of  your  complexity? 

a 

b 

c 

d 

e 

36  . 

Are  the  right  kind  of  quality  assurance 
personnel  assigned  to  your  program? 

a 

b 

c 

d 

e 

1  19 


TO  WHAT  EXTENT 


lot  V«ry  To  To  *  To  m 
•t  all  Little  Some  Great  Very  Crt 
■stent  latent  latent 


37.  Has  your  prime  contractor  instituted 
an  active  quality  improvement  program 

based  upon  continuous  evaluation?  a  b 

38.  Does  your  prime  contractor  conduct 
subcontract  quality  improvement 

programs?  a  b 

TO  WHAT  EXTENT 


Questions  39-42: 

Have  design/manuf actur ing  problems 
impacted  the  following  on  your  system: 


lot  Tory 
at  all  Little 


39  . 

Schedule? 

a 

b 

40. 

System/item  cost? 

a 

b 

41  . 

Per f ormance? 

a 

b 

42. 

Rel iabil ity? 

a 

b 

TO 

WHAT  EXTENT 

43. 

Are  "measurable"  quality  improvement 
goals  defined  by  your  organization? 

a  b 

44  . 

Is  your  contractor's  performance 
measured  against  quality  standards 
than  you  both  agree  upon? 

a 

b 

45. 

Does  your  organization  expend  sufficient 
resources  to  improve  quality? 

a 

b 

46. 

Is  producing  a  quality  product  for  your 
user  the  most  important  organizational 
objective? 

a 

b 

47. 

Does  your  contractor  share  your  quality 
objectives? 

a 

b 

48  . 

Do  you  think  you  have  the  right  number  of 
quality  assurance  personnel  assigned  to 
your  program? 

a 

b 

49. 

Has  the  staff  quality  organization 
aided  in  helping  raise  the  level  of 
quality  in  your  system/item? 

a 

b 

c  d  e 


c  d  e 


To  To  a  To  a 
Some  Great  Very  crt 
latent  latent  latent 

c  d  e 

c  d  e 

c  d  e 

c  d  e 


c  d  e 


c  d  e 

c  d  e 

c  d  e 

c  d  e 

c  d  e 

c  d  e 
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TO  WHAT  EXTENT 

50.  Has  HQ  APSC  leadership  contributed  to 
raising  the  level  of  quality  in  your 

■ot 

at  all 

Vary 

Little 

TO 

Soae 

Extent 

TO  a 
Great 
Extent 

To  a 
Tory  Grt 
Extent 

system/item? 

a 

b 

c 

d 

e 

Questions  51-55: 

To  what  extent  are  you  using  the  following 


51 . 

measures  and/or  indicators  of  quality? 

Comparison  of  planned  versus  actual 

■ot 
at  all 

Very 

Little 

To 

Soae 

Extent 

To  a 
Great 
Extent 

TO  • 
▼ery  Gr 
Extent 

52. 

man-hours  data. 

A  development  process  that  designs 

a 

b 

c 

d 

e 

quality  into  the  product. 

a 

b 

c 

d 

e 

53. 

Monitoring  defects  and  workmanship  data. 

a 

b 

c 

d 

e 

54  . 

Field  performance  data. 

a 

b 

c 

d 

p 

55. 

Subcontractor  yield  rates. 

a 

b 

c 

d 

e 
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Bot 

▼ary 

TO 

To  a 

To  a 

at  ill 

Little 

Boa* 

Croat 

Tory  Crt 

62. 

Producibility  risk  reduction  efforts 
funded  well  before  production.  a 

b 

Bxtant 

c 

Bxtant 

d 

Bstont 

e 

63. 

Hardware  quality  audits.  a 

b 

c 

d 

e 

64  . 

Educational  programs  aimed  at  excellence 
in  management  and  technical  fields.  a 

b 

c 

d 

e 

65. 

Monitor  performance  to  goals.  a 

b 

c 

d 

e 

Questions  66-70: 

To  what  extent  are  the  following 
approaches  to  achieve  better  quality 
being  used  by  your  contractor? 


■ot 

at  all 


Vary  To 
tilttla  lose 
Extent 


To  a  To  a 
Croat  Vary  Crt 
Bxtant  Extant 


66  . 

Total  quality  management  approach. 

a 

b 

c 

d 

e 

67. 

Quality  improvement 

through  design  teams. 

a 

b 

c 

d 

e 

68  . 

Statistical  process 
vendors . 

control  including 

a 

b 

c 

d 

e 

69.  Preferred  vendor  program  with  ratings  on 

quality  and  schedule.  a  b 

70.  Emphasis  on  supplier's  quality.  a  b 


e 

e 


From  the  following  list  of  10  items  pick  the  three  most 
important  contributors  in  improving  quality  and  productivity 
and  rank  them.  Mark  question  153  as  the  most  important 
contributor,  154  as  the  next  most  important  contributor,  and 
155  as  the  third  most  important  contributor.  For  example,  if 
you  think  program  stability  (item  h)  is  most  important  then  you 
would  mark  question  153h,  the  next  item  would  go  in  question 
154,  etc. 

a.  Better  job  of  stating  and  controlling  requirements. 

b.  Contractor  using  transition  templates  to  do  real  risk  management. 

c.  Top-down  corporate  training  to  cultivate  awareness  of  QK  impact  to 
life  cycle  costs. 

d.  Commitment  to  not  accepting  poor  quality. 

e.  Past  performance  rating  in  source  selection  on  quality. 

f.  Require  all  discrepancies  be  referred  to  standards. 

g.  Positive  re-enforcement  programs  (incentives). 

h.  Program  stability  ( f und ing/r equ i rements/des ign) . 

i.  Early  producibi 1 ity/manuf actu r ing/qual ity  input  to  design. 

j.  Disallow  all  costs  for  scrap  and  rework  costs  in  pricing  items. 
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From  the  following  list  of  10  items  pick  the  three  most 
important  items  that  detracts  from  achieving  quality  and 
productivity  and  rank  them.  Mark  question  156  as  the  most 
important  detractor  157  as  the  next  most  important  detractor 
and  158  as  the  third  most  important  detractor. 


a.  Unrealistic  program  schedules. 

b.  Nebulous  definition  of  quality  assurance. 

c.  Lack  of  cooperation  among  personnel  (i.e.,  design,  quality, 
test,  manufacturing,  etc.). 

d.  Old  line  quality  approaches  (i.e.,  inspection  quality). 

e.  Complexity  of  DoD  organization  which  limits  flexibility  and 
response  time. 

f.  Priority  of  quality  within  AF  program  objectives. 

g.  Program  instability. 

h.  Reassigment  of  key  program  personnel. 

i.  Poor  vendor/subcontractor  quality  levels. 

j.  Inadequate  investment  up  front. 

THANK  YOU  AGAIN  POR  YOOR  PARTICIPATION! 


COMMENTS? 
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APPENDIX  E 

QUALITY  ASSURANCE  PERSONNEL 
QUESTIONNAIRE 


QUEST/OA/A/A/ftE 


MAJ  JOHN  A.  CAMPBELL 

Airpower  Research  Institute 

Maxwell  Air  Force  Base  AL 
36112 


Survey  Control  Number  USAF  SCN  87  -  130A 
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INSTRUCTIONS 


1.  Thank  you  for  taking  the  time  to  fill  out  this  questionnaire. 
Most  of  the  answers  will  require  you  to  respond  on  a  five  point 
scale.  Since  there  are  no  "right"  or  "wrong"  answers  and  many 
questions  ask  for  your  opinion,  please  mark  the  answer  that  most 
nearly  represents  your  position. 

2.  You  can  use  this  booklet  to  mark  your  answers  and  then 
transfer  them  to  the  answer  sheet  provided.  Please  use  a  black, 
soft  lead  pencil  (No.  2)  on  the  answer  sheet.  When  you  are  done 
please  return  the  answer  sheet  in  the  enclosed  addressed 
envelope.  if  you  would  like  to  provide  additional  information, 
either  enclose  an  additional  sheet  of  paper  with  your  comments  or 
make  your  comments  on  the  questionai re. 

3.  Please  try  to  answer  each  question,  however  if  no  answer 
applies  or  you  do  not  have  the  knowledge  to  answer  it,  then  skip 
it  and  go  to  the  next  question. 

4.  Your  name  or  organization  is  not  needed  for  this  study. 
However,  if  you  would  like  to  receive  a  brief  summary  of  the 
results  of  this  study,  please  provide  your  name  and  address.  You 
or  your  organization  will  not  in  any  way  be  identified  in  this 
study . 

5.  Thank  you  for  your  time  and  cooperation. 


JOHN  A.  CAMPBELL,  Maj,  USAF 
HQ  AFSC  Research  Fellow 
Airpower  Research  Institute 
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WORK  FORCE  PROFILE 


1  14 

13 

12 

11 

<  09 

1. 

What  is  your  civilian  rank? 

a 

b 

c 

Q 

e 

LTC 

MAJ 

CPT 

1LT 

2LT 

2. 

What  is  your  military  rank? 

a 

b 

c 

d 

e 

3. 

How  many  years  have  you  worked  in 

_>  18 

17-13 

12-8 

7-4 

1  3 

quality  assurance? 

a 

b 

c 

d 

e 

4. 

How  many  different  systems/items 

_>  5 

4 

3 

2 

1 

have  you  worked  on? 

a 

b 

c 

d 

e 

5. 

How  many  years  have  you  worked  on  this 

_>  5 

4 

3 

2 

1 

program? 

a 

b 

c 

d 

e 

6. 

Are  you  dedicated  to  a  system/subsystem 

YES 

NO 

versus  assigned  to  staff? 

a 

b 

7. 

Your  system/item  is  primarily  in  which 

Concep 

Val 

PSD 

Prod 

Deploy 

acquistion  phase? 

a 

b 

c 

d 

e 

8. 

How  many  full-time  quality  personnel 

>  4 

3 

2 

1 

0 

are  assigned  to  your  organization? 

a 

b 

c 

d 

e 

9. 

How  many  weeks  of  ...  .raining  in 

>  12 

11-9 

8-6 

5-3 

<  2 

quality  assurance  v  ye  you  had? 

a 

b 

c 

d 

e 

10. 

How  many  courses  that  taught  statistical 

quality/process  control  have  you  attended  >4 

3 

2 

1 

0 

within  thr  last  5  years? 

a 

b 

c 

d 

e 

11. 

How  many  courses  to  improve  product 

performance  and  durability  have  you 

>  4 

3 

2 

1 

0 

attended  within  the  last  2-3  years  ? 

a 

b 

c 

d 

e 

HS 

HS+ 

Bach 

Bach+ 

Master 

12. 

What  is  your  highest  level  of  education? 

a 

b 

c 

d 

e 

ENGR 

SCI 

BOS 

ARTS 

OTHER 

13. 

What  was  your  college  major? 

a 

b 

c 

d 

e 

14. 

Do  you  have  adequate  opportunities  to 

YES 

NO 

attend  formal  training  courses? 

a 

b 
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Questions  15-24 

Use  the  scale  at  tight  to  answer  questions 


15- 

if 

our 

24.  These  questions  will  help  us  determine 
the  correct  subject  material  is  taught  in  Rot 

formal  courses.  #t 

Very 

Little 

To 

Soae 

Extent 

To  a 
Great 
Extent 

To  a 
Very  Grt 
Extent 

15  . 

To  what  extent  have  formal  courses  helped 
you  perform  your  job? 

a 

b 

c 

d 

e 

QUESTIONS  16-23 

To  what  extent  were  the  following 
subjects  covered  in  formal  courses? 

lot 

at  all 

Very 

Little 

TO 

Soae 

Extent 

To  a 
Great 
Extent 

To  a 
Very  Grt 
Extent 

16. 

Manufacturing  processes. 

a 

b 

c 

d 

e 

17. 

Special  quality  assurance  processes. 

a 

b 

c 

d 

e 

18. 

Product  and  hardware  familiarization. 

a 

b 

c 

d 

e 

19. 

State-of-the-art  processing  technology. 

a 

b 

c 

d 

e 

29. 

Computer  systems  including  software. 

a 

b 

c 

d 

e 

21. 

Cost  of  quality  concepts. 

a 

b 

c 

d 

e 

22. 

Quality  contract  requirements. 

a 

b 

c 

d 

e 

23. 

Testing  procedures. 

a 

b 

c 

d 

e 

24. 

How  important  do  you  think  these  topics 
are  to  you  in  performing  your  job? 

a 

b 

c 

d 

e 

SSSSaxS3a33S33asaa33333333a3333S3333a3a333=S= 

SYSTEM  PROGRAM  QUALITY 

Use  the  scale  at  right  to  answer 
questions  25-28. 

3S33S3S 

Daily 

33SSSS 

Weekly 

3393333; 

Monthly 

1333333333333 

Qtrly  Never 

25. 

How  often  do  you  interface  with  the 

SPO  program  management  personnel? 

a 

b 

c 

d 

e 

26. 

How  often  do  you  review  user  complaint 
data/trends  on  your  system/item? 

a 

b 

c 

d 

e 

27. 

How  often  do  you  review  your 
contractor's  rework,  repair,  retest 
scrap,  etc.,  levels? 

a 

b 

c 

d 

e 

28.  How  often  do  you  review  your 

contractor's  "cost  of  quality"  (failure 
costs  +  routine  inspect  ion  + 


a 


b 


c 


d 


e 


33333353S3S33333X333S333333333333333333333333 

29.  How  many  quality  of  conformance  problems 
have  had  significant  program  impacts 
during  the  past  year  (i.e.,  schedule, 
cost,  accident,  publicity,  etc.)? 

3  3  3  3  3 

>10 

a 

33333333 

5 

b 

3  3  3  3  3  3  3 

3 

c 

3  3  3  3  3  3  3 

1 

d 

0 

e 

30.  As  a  percentage  of  total  contract  value, 
what  would  you  estimate  the  "cost  > 

of  quality"  is  for  your  program? 

25% 

a 

24-20% 

b 

19-15% 

c 

14-10% 

d 

<  9% 

e 

Use  the  scale  at  right  to  answer  questions 
31-33.  These  questions  are  asking  you  to 
rate  the  breakdown  of  the  cost  of  quality. 

If  you  don't  know,  leave  it  blank. 

90% 

89-70% 

69-50% 

3  3  3  3  3  3  3 

49-30% 

<  29% 

31.  What  percentage  does  failure  cost 
contribute  to  your  cost  of  quality? 

a 

b 

c 

d 

e 

32.  What  percentage  does  routine  factory 
inspection  contribute  to  your  cost 
of  quality? 

a 

b 

c 

d 

e 

33.  What  percentage  does  prevention  measures 
contribute  to  your  cost  of  quality? 

a 

b 

c 

d 

e 

34.  What  degree  of  confidence  do  you  have 

in  the  accuracy  for  your  cost  of  > 

quality  estimate? 

95% 

a 

94-75% 

b 

74-50% 

c 

49-25% 

d 

<  25% 
e 

333333  =  33333333333-r33333333333333333333333*S3333333333333333333333333333333  =  3 

The  next  qroup  of  questions  will  help  us  determine  from  your  perspective 
how  best  to  improve  quality  assurance  management.  Each  question  is 
asking  to  what  extent  you  think  the  statement  applies  to  you/your 
organization.  Therefore,  each  question  should  be  preceded  by  TO  WHAT 

EXTENT.  If  you  don't  know  the  answer,  leave  it  blank. 

Use  the  scale  at  right  to  answer  >ot  very  To  To  .  To  . 

questions  35-83  .  *n  Little  soee  Great  Very  Grt 

latent  latent  latent 

35.  Does  your  user  influence  contractor 

quality  of  hardware/services?  a  b  c  d  e 

36.  Have  delivery  schedules  taken  priority 
over  quality  decisions? 

a 

b 

c 

d 

e 

37.  Do  engineering  errors  lead  to  problems 
in  managing  your  system/item? 

a 

b 

c 

d 

e 

38.  Do  manufacturing  errors  lead  to  problems 
in  managing  your  system/item? 

a 

b 

c 

d 

e 
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tot  very  To  To  »  To  • 

•t  all  Little  Soae  Great  Very  Grt 

TO  WHAT  EXTENT  Extent  Extent  Extent 

39.  Do  quality  assurance  personnel  help 
achieve  the  expected  level  of  quality 

on  your  program?  a  b  c  d  e 

40.  Are  you  able  to  influence  the  contractor 
to  provide  quality  products  using 

contract  requirements?  a  b  c  d  e 

41.  Are  you  able  to  influence  the  contractor 
to  provide  quality  products  using  good 

business  practices?  a  b  c  d  e 

42.  Would  a  Pirm  Fixed  Price  type 
contract  influence  your  contractor 

to  provide  a  quality  product?  abode 

43.  Would  an  award  fee  influence  your 

contractor  to  provide  a  quality  product?  a  b  c  d  e 

44.  Would  a  warranty  influence  your  contractor 

to  provide  a  quality  product?  abode 

45.  Has  your  contractor  used  employee 
motivation  techniques  to  improve 

quality  of  products?  abode 

46.  H is  your  contractor's  top  management's 
attention  influenced  the  quality 

of  the  product  for  your  user?  a  b  c  d  e 

47.  Do  you  focused  on  defect  prevention 

rather  than  detection?  a  b  c  d  e 

48.  Do  you  think  the  number  of  quality 
problems  in  the  last  year  *  -e  normal 

for  a  program  of  your  complexity?  abode 

49.  Are  the  right  kind  of  quality  assurance 

personnel  assigned  to  your  program?  a  b  c  d  e 

50.  Has  your  prime  contractor  instituted 
an  active  quality  improvement  program 

based  upon  continuous  evaluation?  a  b  c  d  e 

51.  Does  your  prime  contractor  conduct 
subcontract  quality  improvement 

programs?  a  b  c  d  e 
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Rot  Very  To  To  a  To  a 

at  all  Little  Soie  Great  Very  Grt 

TO  WHAT  EXTENT  Extent  latent  latent 
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Questions  52-55: 

Have  design/manuf acturing  problems 
impacted  the  following  on  your  system: 


52  . 

Schedule? 

a 

b 

c 

d 

e 

53. 

System/item  cost? 

a 

b 

c 

d 

e 

54  . 

Performance? 

a 

b 

c 

d 

e 

55. 

Reliability? 

a 

b 

c 

d 

e 

TO 

WHAT  EXTENT 

Rot 

at  all 

Very 

Little 

TO 

Soae 

latent 

To  a 
Great 
latent 

21=  3  3  3  3 

To  a 

Very  Grt 
latent 

56. 

Are  "measurable"  quality  improvement 
goals  defined  by  your  organization? 

a 

b 

c 

d 

e 

57. 

Is  your  contractor's  performance 
measured  against  quality  standards 
that  you  both  agree  upon? 

a 

b 

c 

d 

e 

58. 

Does  your  organization  expend  sufficient 
resources  to  improve  quality? 

a 

b 

c 

d 

e 

59. 

Is  producing  a  quality  product  for  your 
user  the  most  important  organizational 
objective? 

a 

b 

c 

d 

e 

60  . 

Does  your  contractor  share  your  quality 
objectives? 

a 

b 

c 

d 

e 

61 . 

Do  you  think  you  have  the  right  number  of 
quality  assurance  personnel  assigned  to 
your  program? 

a 

b 

c 

d 

e 

62. 

Has  the  staff  quality  organization 
aided  in  helping  raise  the  level  of 
quality  in  your  system/item? 

a 

b 

c 

d 

e 

63. 

Has  HQ  APSC  leadership  contributed  to 
raising  the  level  of  quality  in  your 
system/item? 

a 

b 

c 

d 

e 
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Questions  64-68: 


64. 

To  what  extent  are  you  using  the  following 
measures  and/or  indicators  of  quality? 

Comparison  of  planned  versus  actual 

Rot 

tt  All 

Very 

Little 

TO 

Soae 

Extent 

To  a 
Great 
Extent 

TO  a 

very  Grt 
Extent 

man-hours  data. 

a 

b 

c 

d 

e 

65. 

A  development  process  that  designs 
quality  into  the  product. 

a 

b 

c 

d 

e 

66. 

Monitoring  defects  and  workmanship  data. 

a 

b 

c 

d 

e 

67. 

Field  performance  data. 

a 

b 

c 

d 

e 

68. 

Subcontractor  yield  rates. 

a 

b 

c 

d 

e 

Questions  69-73: 

To  what  extent  does  your  contractor  use  lot  Very  to  to  a  to  • 
the  following  measures  and/or  indicators  at  all  Little  soae  Great  Very  Grt 
of  quality?  Extent  Extent  Extent 


69.  Product  return  rate  (cost  to  rework). 

a 

b 

c 

d 

e 

70.  Degrees  of  manufacturing  process 

standardization . 

a 

b 

c 

d 

e 

71.  Production  yield  rates. 

a 

b 

c 

d 

e 

72.  Accept /re ject  rate  at  work  centers. 

a 

b 

c 

d 

e 

73.  MRB/QDR  action  effectiveness  (repeats). 

a 

b 

c 

d 

e 

rassass 

S33S=33SSEae33S3 

xssaxas 

Questions  74-78: 

To  what  extent  does  your  SPO  use 
the  following  approaches  to  achieve 
better  quality? 

Rot 

at  all 

Very 

Little 

TO 

Soae 

Extent 

To  a 
Great 
Extent 

To  a 

Very  Grt 
Extent 

74.  Team  building  between  SPO,  CAS,  and 

contractor . 

a 

b 

c 

d 

e 

75.  Producibility  risk  reduction  efforts 

funded  well  before  production. 

a 

b 

c 

d 

e 

76.  Hardware  quality  audits. 

a 

b 

c 

d 

e 

77.  Educational  programs  aimed  at  excellence 

in  management  and  technical  fields. 

a 

b 

c 

d 

e 

78.  Monitor  performance  to  goals. 

a 

b 

c 

d 

e 
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Questions  79-83: 


To  what  extent  are  the  following 
approaches  to  achieve  better  quality 
being  used  by  your  contractor? 

Mot 

at  all 

Very 

Little 

TO 

Soae 

latent 

To  a 
Creat 
latent 

To  a 

Tory  Crt 

litent 

79. 

Total  quality  management  approach. 

a 

b 

c 

d 

e 

80. 

Quality  improvement  through  design  teams. 

a 

b 

c 

d 

e 

81. 

Statistical  process  control  including 
vendors . 

a 

b 

c 

d 

e 

82  . 

Preferred  vendor  program  with  ratings  on 
quality  and  schedule. 

a 

b 

c 

d 

e 

8  3  . 

Emphasis  on  supplier's  quality. 

a 

b 

c 

d 

e 

From  the  following  list  of  10  items  pick  the  three  most 
important  contributors  in  improving  quality  and  productivity 
and  rank  them.  Mark  question  153  as  the  most  important 
contributor,  154  as  the  next  most  important  contributor,  and 
155  as  the  third  most  important  contributor.  For  example,  if 
you  think  program  stability  (item  h)  is  most  important  then  you 
would  mark  question  153h,  the  next  item  would  go  in  question 
154,  etc. 


a.  Better  job  of  stating  and  controlling  requirements. 

b.  Contractor  using  transition  templates  to  do  real  risk  management. 

c.  Top-down  corporate  training  to  cultivate  awareness  of  QA  impact  to 
life  cycle  costs. 

d.  Commitment  to  not  accepting  poor  quality. 

e.  Past  performance  rating  in  source  selection  on  quality. 

f.  Require  all  discrepancies  be  referred  to  standards. 

g.  Positive  re-enforcement  programs  (incentives). 

h.  Program  stability  (funding/requirements/design). 

i.  Early  producibility/manufacturing/quality  input  to  design. 

j.  Disallow  all  costs  for  scrap  and  rework  costs  in  pricing  items. 
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From  the  following  list  of  18  items  pick  the  three  most 
important  items  that  detracts  from  achieving  quality  and 
productivity  and  rank  them.  Mark  question  156  as  the  most 
important  detractor  157  as  the  next  most  important  detractor 
and  158  as  the  third  most  important  detractor. 

a.  Unrealistic  program  schedules. 

b.  Nebulous  definition  of  quality  assurance. 

c.  Lack  of  cooperation  among  personnel  (i.e.,  design,  quality, 
test,  manufacturing,  etc.). 

d.  Old  line  quality  approaches  (i.e.,  inspection  quality). 

e.  Complexity  of  DoD  organization  which  limits  flexibility  and 
response  time. 

f.  Priority  of  quality  within  AF  program  objectives. 

g.  Program  instability. 

h.  Reassigment  of  key  program  personnel. 

i.  Poor  vendor/subcontractor  quality  levels. 

j.  Inadequate  investment  up  front. 

THANK  YOU  AGAIN  FOR  YOUR  PARTICIPATION! 


COMMENTS? 
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APPENDIX  F 

PROGRAM  DIRECTOR’S  DATA 


WORK  FORCE  PROFILE 


Number (Percentage) 


1. 

GS/M 

What  is  your  military/civilian  MIL 

equivalent  rank? 

16 

B/G 

7(12) 

15 
Col 
43 (71) 

14 

LTC 

10(16) 

13  < 
MAJ 

12 

<  CPT 
1(2) 

2, 

How  many  years  have  you  worked  in 
a  program  management  function? 

>  10 
43(71) 

9-8 

5(8) 

7-6 

3(5) 

5-4 

3(5) 

<  3 
?(10) 

3. 

How  many  different  systems/items 
have  you  managed? 

>  5 

36(59) 

4 

8(13) 

3 

9(15) 

2 

4(7) 

1 

4(7) 

4. 

How  many  years  have  you  worked  on  this 
program  (your  current  one)? 

>  5 

1(8) 

4 

5(8) 

3 

8(13) 

2 

13(21) 

1 

30(49) 

5. 

Are  you  the  program  director /program 
manager  versus  a  project  manager  on 
a  system/subsystem? 

YES 

61 (100) 

NO 

6. 

Your  system/item  is  primarily  in  which 
acquisition  phase? 

Concep 

2(3) 

Val 

2(3) 

PSD 

28(46) 

Prod 

25(41) 

Deploy 

4(7) 

7. 

How  many  full-time  quality  personnel 
are  assigned  to  your  organization? 

>  4 

20 (33) 

3 

3(5) 

2 

10(16) 

1 

18(30) 

0 

10(16) 

8. 

How  much  of  your  day  (hours)  is  spent 
on  quality  issues  (failures/corrections) 

>  5 

4-3 

4(7) 

2 

10(16) 

1 

29(48) 

0 

17(28) 

9. 

How  many  weeks  of  AF  training  in 
program  management  have  you  had? 

>  10 

43(71) 

9-8 

2(3) 

7-6 

2(3) 

5-4 

5(8) 

<  3 
9(15) 

10  . 

How  many  hours  of  training/education 
have  you  had  specifically  addressing  how 
to  improve  quality  hardware/services? 

>  40 

■5(15) 

39-30 

3(5) 

29-20 

11(18) 

19-10 

11(18) 

<  9 

26(43) 

11 . 

How  many  courses  to  improve  product 
performance  and  durability  have  you 
attended  within  the  last  2-3  years  ? 

>  4 

2(3) 

3 

2(3) 

2 

2(3) 

1 

11(18) 

0 

44(72) 

ICR 

3S33333333S3333333S3333333333333S33S3S33S 

IS  3333  3 

333333 

SYSTEM  PROGRAM  QUALITY 

Use  the  scale  at  right  to  answer 

questions  12-15.  Rot  An«v«d  Daily  Weakly  Monthly  Qtrly  Reeer 


12.  How  often  do  you  interface  directly 
with  the  SPO  personnel  who  track  quality 

performance?  29(48)  24(39)  5(8)  3(5) 

13.  How  often  do  you  personally  review 
user  complaint  data/trends  on  your 

system/item?  2(3)  5(8)  27(44)  16(26)  8(13)  3(5) 
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Not  Ansved  Dally  Weekly  Monthly  Qtrly  never 


14.  How  often  do  you  personally  review 

your  contractor's  rework,  repair, 
retest  scrap,  etc.,  levels?  2(1) 

15.  How  often  do  you  personally  review 
your  contractor's  "cost  of  quality" 

( failure  costs  +  routine  inspec- 
tion  +  prevention  costs)  data/trends? 

2(3) 


9(15)  22(36)  15(25)  12(20) 


4(7)  12(20)  16(26)  26(43) 


16.  How  many  quality  of  conformance  problems 

have  had  significant  program  impacts 
during  the  past  year  (i.e.,  schedule,  >10 
cost,  accident,  publicity,  etc.)?  2(3)  5(8) 

17.  As  a  percentage  of  total  contract  value, 

what  would  you  estimate  the  "cost  >  25% 

of  quality"  is  for  your  program?  4 ( 7 ) 


5  3  10 

9(15)  17(28)  7(12)  19(31) 


24-20%  19-15%  14-10%  <  9% 

8(13)  11(18)  17(28)  20(33) 


Use  the  scale  at  right  to  answer  questions 
18-20.  These  questions  are  asking  you  to 
rate  the  breakdown  of  the  cost  of  quality. 

If  you  don't  know,  leave  it  blank.  BK  >  90%  89-70%  69-50%  49-30%  <  29% 


18.  What  percentage  does  failure  cost 
contribute  to  your  cost  of  qua  1 i ty ?35  ( 5 7  ) 

19.  What  percentage  does  routine  factory 
inspection  contribute  to  your  cost 

of  quality?  35(57) 

20.  What  percentage  does  prevention  measures 
contribute  to  your  cost  of  quality?35 ( 57 )  1(2) 


4(7)  7(12)  13(21) 


3(5)  3(5)  5(8)  11(18) 


2(3)  3(5)  17(281 


21.  What  degree  of  confidence  do  you  have 

in  the  accuracy  for  your  cost  of  BK  >  95%  94-75%  74-50%  49-25%  £  25% 

quality  estimate?  31(51)  1(2)  6(10)  7(12)  3(5)  11(18) 


The  next  qroup  of  questions  will  help  us  determine  from  your  perspective 
how  best  to  improve  quality  assurance  management.  Each  question  is 
asking  to  what  extent  you  think  the  statement  applies  to  you/your 
organization.  Therefore,  each  question  should  be  preceded  by  TO  WHAT 
EXTENT.  If  you  don't  know  the  answer,  leave  it  blank. 

Use  the  scale  at  right  to  answer  „ot  *ot  T,ry  To  To  ,  To  , 

questions  22-70  .  Answered  at  all  Little  Sowe  Great  very  Grt 

Extent  Extent  Extent 

22.  Does  your  user  influence  contractor 

quality  of  hardware/services?  3(5)  3(5)  10(16)  20(33)  19(31)  6(10) 
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Hot 

Not 

Very 

TO 

To  a 

Answered 

at  all 

Little 

Some 

Great 

TO 

WHAT  EXTENT 

Extent 

Extent 

23  . 

Have  delivery  schedules  taken  priority 
over  quality  decisions?  2(3) 

10(16) 

23(38) 

19(31) 

6(10) 

24. 

Do  engineering  errors  lead  to  problems 
in  managing  your  system/item? 

2(3) 

14(23) 

33(54) 

7(12) 

25  . 

Do  manufacturing  errors  lead  to  problems 
in  managing  your  system/item?  2(3) 

4(7) 

12(20) 

28(46) 

12(20) 

26. 

Do  quality  assurance  personnel  help  you 
achieve  the  expected  level  of  qualify? 

1(2) 

5(8) 

30(49) 

21(34) 

27  . 

Are  you  able  to  influence  the  contractor 
to  provide  quality  products  using 
contract  requirements? 

4(7) 

16(26) 

29(48) 

28. 

Are  you  able  to  influence  the  contractor 
to  provide  quality  products  using  good 
business  practices? 

1(2) 

2(3) 

25 (41) 

22(36) 

29. 

would  a  Firm  Fixed  Price  type 
contract  influence  your  contractor 
to  provide  a  quality  product?  4(7) 

7(12) 

21(34) 

23(38) 

4(7) 

30. 

Would  an  award  fee  influence  your 
contractor  to  provide  a  quality  product? 

8(13) 

19(31) 

25(41) 

2(3) 

31  . 

would  a  warranty  influence  your  contractor 

to  provide  a  quality  product?  4(7) 

1(2) 

5(8) 

22(36) 

26(43) 

32. 

Has  your  contractor  used  employee 
motivation  techniques  to  improve 
quality  of  products?  5(8) 

5(8) 

28(46) 

18(30) 

33. 

Has  your  contractor's  top  management's 
attention  influenced  the  quality 
of  the  product  for  your  user?  2(3) 

4(7) 

22(36) 

27(44) 

34  . 

Do  your  quality  personnel  focus  on  defect 
prevention  rather  than  detection?  4(7) 

11(18) 

23(38) 

17(28) 

35. 

Do  you  think  the  number  of  quality 
problems  in  the  last  year  were  normal 
for  a  program  of  your  complexity?  4(7) 

2(3) 

7(12) 

18 (30) 

19(31) 

36. 

Are  the  right  kind  of  quality  assurance 
personnel  assigned  to  your  program?2(3) 

7(12) 

4(7) 

22(36) 

18(30) 

To  a 

Very  Grt 
Extent 

4(7) 

1(2) 

3(5) 

11(18) 

11(18) 

2(3) 

7(12) 

3(5) 

5(8) 

6(10) 

6(10) 

9(15) 

8(13) 
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TO 

Rot 

WHAT  EXTENT  Anavered 

Rot 

at  all 

vary 

Little 

TO 

Soae 

Extent 

To  a 
Great 
Extent 

To  a 

Very  Grt 
Extent 

37 

.  Has  your  prime  contractor  instituted 

an  active  quality  improvement  program 
based  upon  continuous  evaluation?  8(13) 

9(15) 

18 (30) 

18(30) 

7(12) 

38 

.  Does  your  prime  contractor  conduct 

subcontract  quality  improvement 
programs?  8(13) 

6(10) 

25 (41) 

17(28) 

4(7) 

TO 

WHAT  EXTENT 

=  3=  : 

=c3£33acsa2sas33s3sssssssss  =  sa  =  xssss  =  =  ssss3  =  s;ss:s 

11 

11 

II 

II 

II 

II 

II 

====== 

Hot 

Rot 

Very 

TO 

To  a 

To  a 

Answered 

at  all 

Little 

Some 

Great 

Very  Grt 

Questions  39-42: 

Extent 

Extent 

Extent 

Have  design/manufacturing  problems 
impacted  the  following  on  your  system: 

39 

.  Schedule?  1(2) 

4(7) 

7(12) 

19(31) 

14(23) 

14(23) 

40 

.  System/item  cost?  1(2) 

6(10) 

12(20) 

22(36) 

13(16) 

8(13) 

41 

.  Performance?  1(2) 

7(12) 

21(34) 

16(26) 

10(16) 

4(7) 

42 

.  Reliability?  1(2) 

10(16) 

13(21) 

19(31) 

12(20) 

4(7) 

TO 

WHAT  EXTENT 

3  =  : 

3333333333333333*333*3333333333333333333333 

33323333333333 

»»■* == 

3333333 

3  3  3  3  3  3 

43 

.  Are  "measurable"  quality  improvement 

goals  defined  by  your  organization?  6(10) 

4(7) 

19(31) 

17(28) 

8(13) 

5(8) 

44 

.  is  your  contractor's  performance 

measured  against  quality  standards 
that  you  both  agree  upon?  5(8) 

6(10) 

26(43) 

15(25) 

7(12) 

45 

.  Does  your  organization  expend  sufficient 

resources  to  improve  quality?  5(8) 

2(3) 

7(12) 

22(36) 

19(31) 

4(7) 

46 

.  Is  producing  a  quality  product  for  your 

user  the  most  important  organizational 
objective?  1(2) 

1(2) 

2(3) 

10 (16) 

23(38) 

22(36) 

47 

.  Does  your  contractor  share  your  quality 

objectives?  2(3) 

1(2) 

22(36) 

22(36) 

12(20) 

48 

.  Do  you  think  you  have  the  right  number  of 

quality  assurance  personnel  assigned  to 
your  program?  1(2) 

13(21) 

15(25) 

14(23) 

12(20) 

4(7) 

49 

.  Has  the  staff  quality  organization 

aided  in  helping  raise  the  level  of 
quality  in  your  system/item?  1(2) 

9(15) 

22(36) 

19(31) 

6(10) 

2(3) 
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TO  WHAT  EXTENT 


*ot  Hot  Very  To  To  a  To  a 

Answered  at  ail  Little  Soae  Great  Vary  Grt 

Bxtent  Extant  Extant 

50.  Has  HQ  APSC  leadership  contributed  to 
raising  the  level  of  quality  in  your 

system/item?  19(31)  20(33)  17(28)  3(5)  1(2) 


KS3  =  3333X3SS:S3S;S=:S=SSS33SSaa(SS3S  =  3:sa33  =  3SS3S233S33a:SS33333S3S  =  3  =  S332  =  =  33S  =  S  = 

Questions  51-55: 

To  what  extent  are  you  using  Hot  Bot  Very  To  To  a  To  a 

the  following  measures  and/or  Anawarad  at  all  Little  Soae  Great  Vary  Grt 

indicators  of  quality?  Extant  Extant  Extant 


51.  Comparison  of  planned  versus  actual 
man-hours  data. 


7(12)  6(10)  10(16)  21(34)  10(16)  5(8) 


52.  A  development  process  that  designs 

quality  inco  the  product.  5(8)  2(3)  4(7)  15(25)  20(33)  14(23) 

53.  Monitoring  defects  and  workmanship  data.  2(3)  2(3)  20(33)  22(36)  7(12) 

6(10) 

54.  Field  performance  data.  9(15)  7(12)  1(2)  12(20)  20(33)  10(16) 


55.  Subcontractor  yield  rates, 


9(15)  8(13)  12(2)  19(31)  8(13)  3(5) 


Questions  56-60: 

To  what  extent  does  your  contractor 
use  the  following  measures  and/or  i»ot 
indicators  of  quality?  Anawa 


”  Hot  Not  Very  To  To  a  To  a 

Answered  at  all  Little  Some  Great  Very  Grt 

Extent  Extant  Bxtent 


56.  Product  return  rate  (cost  to  rework). 

7(12) 

15(25) 

17(28) 

4(7) 

16(26) 

57.  Degrees  of  manufacturi 

ng  process 

standardization . 

19(31) 

1(2) 

7(12) 

16(26) 

11(13) 

5(8) 

58.  Production  yield  rates 

.  18(30) 

2(3) 

3(5) 

9(15) 

23(38) 

4(7) 

59.  Accept/re ject  rate  at 

work  centers  .  16  (  26 ) 

1(2) 

4(7) 

14(23) 

20(33) 

4(7) 

60.  Material  review  board/quality  deficiency 
reports  (MRB/QDR)  action  effectiveness 
(repeats).  12(20) 


2(3)  14(23)  23(38)  8(13) 


Questions  61-65: 

To  what  extent  does  your  SPO  use 
the  following  approaches  to 
achieve  better  quality? 


Bot  Not  Very  To  To  a  To  a 

Answered  at  all  Little  Some  Great  Vary  Grt 

Extant  Extent  Extant 


61.  Team  building  between  SPO,  CAS,  and 
contractor . 


1(2)  19(31)  29(48)  11(18) 


143 


Not 

Not 

very 

TO 

TO  a 

To  a 

Answered 

at  all 

Little 

Soae 

Great  Very  Grt 

latent 

Extent 

Extent 

62.  Producibi 1 ity  risk  reduction 

efforts 

funded  well  before  production.  4(7) 

8(13) 

17(28) 

13(21) 

11(18) 

7(12) 

63.  Hardware  quality  audits. 

3(5) 

1(2) 

6(13) 

21(34) 

20(33) 

8(13) 

64.  Educational  programs  aimed  at  excellence 

in  management  and  technical 

fields. 5 (8) 

3(5) 

15(25) 

22(36) 

13(21) 

2(3) 

65.  Monitor  performance  to  goals 

3(5) 

1(2) 

8(13) 

20(33) 

21(34) 

6(10) 

Questions  66-70: 

II 

II 

II 

II 

II 

II 

======= 

3  =  3t  3  3  =  3 

=  =  =  =  =  =  = 

=  =  =  =  =  = 

To  what  extent  are  the  following 

approaches  to  achieve  better 
quality  being  used  by  your 

Not 

Answered 

Not 

at  all 

Very 

Little 

To 

Some 

To  a 
Great 

To  a 

Very  Grt 

contractor? 

Extent 

Extent 

Extent 

66.  Total  quality  management  approach.  13(16) 

1(2) 

4(7) 

22(36) 

18(30) 

5(8) 

67.  Quality  improvement  through 

design  teams. 

1(2) 

8(13) 

22(36) 

13(21) 

3(13) 

8(13) 

68.  Statistical  process  control 

including 

vendors . 

15(25) 

2(3) 

8(13) 

23(38) 

7(12) 

4(7) 

69.  Preferred  vendor  program  with  ratings  on 

quality  and  schedule. 

12(23) 

1(2) 

2(3) 

21(34) 

15(25) 

8(13) 

70.  Emphasis  on  supplier's  quali 

ty.  9(15) 

19(31) 

24(39) 

7(12) 

From  the  following  list  of  10  items  pick  the 

three 

most 

important  contributors  in  improving  quality  and  productivity 


and  rank  them.  Mark  question  153  as  the  most  important 
contributor,  154  as  the  next  most  important  contributor,  and 
155  as  the  third  most  important  contributor.  For  example,  if 
you  think  program  stability  (item  h)  is  most  important  then  you 
would  mark  question  153h,  the  next  item  would  go  in  question 
154,  etc. 

Ranking 


a.  Better  job  of  stating  and  controlling  requirements.  6 

b.  Contractor  using  transition  templates  to  do  real  risk  management.  10 

c.  Top-down  corporate  training  to  cultivate  awareness  of  QA  impact  to 

life  cycle  costs.  4 

d.  Commitment  to  not  accepting  poor  quality.  2 

e.  Past  performance  rating  in  source  selection  on  quality.  7 

f.  Requite  all  discrepancies  be  referred  to  standards.  9 

g.  Positive  re-enforcement  programs  (incentives).  5 

h.  Program  stability  ( f unding/r equi r ement s/design ) .  3 

i.  Early  producibil ity/manuf acturing/quality  input  to  design.  1 

j.  Disallow  all  costs  for  scrap  and  rework  costs  in  pricing  items.  8 
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From  the  following  list  of  10  items  pick  the  three  most 
important  items  that  detracts  from  achieving  quality  and 
productivity  and  rank  them.  Mark  question  156  as  the  most 
important  detractor  157  as  the  next  most  important  detractor 
and  158  as  the  third  most  important  detractor. 

a.  Unrealistic  program  schedules. 

b.  Nebulous  definition  of  quality  assurance. 

c.  Lack  of  cooperation  among  personnel  (i.e.,  design,  quality, 
test,  manufacturing,  etc.). 

d.  Old  line  quality  approaches  (i.e.,  inspection  quality). 

e.  Complexity  of  DoD  organization  which  limits  flexibility  and 
response  time. 

f.  Priority  of  quality  within  AF  program  objectives. 

g.  Program  instability. 

h.  Peessigmant  of  key  program  personnel. 

i.  Poor  vendor/subcontractor  quality  levels. 

j.  Inadequate  investment  up  front. 


Ranking 

4 

7 

1 

6 


10 

5 

2 


SAMPLE  SIZE  -  61 


U>  oo  Vj£> 


APPENDIX  G 

PROGRAM  MANAGER'S  DATA 


WORK  FORCE  PROFILE 


Number (Precentage) 


1 . 

GS/M 

What  is  your  military/civilian  MIL 

equivalent  rank? 

16 

B/G 

3(1) 

15 

Col 

3(1) 

14 

LTC 

40 (19) 

'  *3 

MAJ 

41(19) 

<  12 
<  CPT 
170(60) 

2. 

How  many  years  have  you  worked  in 
a  program  management  function? 

>  10 
26(12) 

9-8 

15(7) 

7-6 

17 

5-4 

42(19) 

<  3 

115(53) 

3  . 

How  many  different  systems/ items 
have  you  managed? 

>  5 

54 (25) 

4 

22(10) 

3 

40(18) 

2 

50(23) 

1 

49(23) 

4  . 

How  many  years  have  you  worked  on  this 
program  (your  current  one)? 

>  5 

5(2) 

4 

13(6) 

3 

47(22) 

2 

63(29) 

1 

86(40) 

5. 

Are  you  the  program  director /program 
manager  versus  a  project  manager  on 
a  system/subsystem? 

YES 

66(30) 

NO 

147  (68) 

6. 

Your  system/item  is  primarily  in  which 
acquisition  phase? 

Concep 

6(3) 

Val 

27(12) 

FSD 

100(46) 

Prod 

48(22) 

Deploy 

31(14) 

7. 

How  many  full-time  quality  personnel 
are  assigned  to  your  organization? 

>  4 

64(30) 

3 

16  (7) 

2 

26(12) 

1 

49(23) 

0 

54(25) 

8. 

How  much  of  your  day  (hours)  is  spent 
on  quality  issues  (failures/corrections) 

>  5 

?  10(5) 

4-3 

23(11) 

2 

33(15) 

1 

71(33) 

0 

73(34) 

9. 

How  many  weeks  of  AF  training  in 
program  management  have  you  had? 

>  10 

“67(31) 

9-8 

17(3) 

7-6 

44(20) 

5-4 

49(23) 

<  3 

40(19) 

10. 

How  many  hours  of  training/education 
have  you  had  specifically  addressing  how 
to  improve  quality  hardware/services? 

>  40 

73 (11) 

39-30 

7(3) 

29-20 

18(8) 

19-10 

29(13) 

<  9 

T38  (64) 

11 . 

How  many  courses  to  improve  product 
performance  and  durability  have  you 
attended  within  the  last  2-3  years  ? 

>  4 

1(1) 

3 

7(3) 

2 

13(6) 

1 

44(20) 

0 

152(70) 

SYSTEM  PROGRAM  QUALITY 

Use  the  scale  at  right  to  answer 

questions  12-15.  Wot  *•»»*•<*  Daily  Weekly  Monthly  Qtrly  Sever 

12.  How  often  do  you  interface  directly 
with  the  SPO  personnel  who  track  quality 

performance?  4(2)  39(19)  66(30)  39(18)  24(11)  43(20) 

13.  How  often  do  you  personally  review 
user  complaint  data/trends  on  your 

system/item?  11(5)  16(7)  41(19)  48(22)  24(11)  72(33) 
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Mot  Antwed  Daily  weekly  Monthly  Qtrly  Mver 


14.  How  often  do  you  personally  review 
your  contractor's  rework,  repair, 

retest  scrap,  etc.,  levels?  11(5)  3(1)  15(7)  36(17)  38(18)  107(49) 

15.  How  often  do  you  personally  review 
your  contractor's  "cost  of  quality" 

( failure  costs  +  routine  ins pec  - 

tion  +  prevention  costs)  data/trends?  9(4)  18(8)  30(14)  139(64) 

15(7) 


16.  How  many  quality  of  conformance  problems 
have  had  significant  program  impacts 

during  the  past  year  (i.e.,  schedule,  BK  >10  5  310 

cost,  accident,  publicity,  etc.)?  20(9)  18(8)  21(10)  41(19)  43(20)  66(30) 

17.  xs  a  percentage  of  total  contract  value, 

what  would  you  estimate  the  "cost  BK  >  25%  24-20%  19-15%  14-10%  <  9% 
of  quality"  is  for  your  program?  35  (  16  )  1 6  (7)  16(7)  18  (8)  51  (24)  71(34) 


Use  the  scale  at  right  to  answer  questions 
18-20.  These  questions  are  asking  you  to 
rate  the  breakdown  of  the  cost  of  quality. 

If  you  don't  know,  leave  it  blank.  BK  90%  89-70%  69-50%  49-30%  <  29% 

18.  What  percentage  does  failure  cost 
contribute  to  your  cost  of  qua  1 i t y ?127  (  59  )  2  (  1)  5  (  2) 

19.  What  percentage  does  rout.ne  factory 
inspection  contribute  to  your  cost 

of  quality?  125(58)  1) 

20.  What  percentage  does  prevention  measures 
contribute  to  your  cost  of  qual  i  ty? 129  (  59 )  1(1) 

21.  What  degree  of  confidence  io  you  have 

in  the  accuracy  for  your  cost  of  BK  >_  95%  94-75%  74-50%  49-25%  <  25% 

quality  estimate?  122(56)  4(2)  8(4)  20(9)  16(7)  ”26(12) 


12(6) 

10(5) 

38(18) 

13(6) 

11(5) 

42 (19) 

8(4) 

15(7) 

41(19) 

The  next  qroup  of  questions  will  help  us  determine  from  your  perspective 
how  best  to  improve  quality  assurance  management.  Each  question  is 
asking  to  what  extent  you  think  the  statement  applies  to  you/your 
organization.  Therefore,  each  question  should  be  preceded  by  TO  WHAT 
EXTENT.  If  you  don't  know  the  answer,  leave  it  blank. 

Use  the  scale  at  right  to  answer 

questions  22-70  .  Mot  Mot  Very  To  To  •  To  • 

Answered  st  ill  Little  Soee  Greet  Very  Grt 
_  Extent  Extent  Extent 

22.  Does  your  user  influence  contractor 

quality  of  hardware/services?  21(10)  17(8)  28(13)  61(28)  66(30)  14(7) 
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Hot  Hot  Very  To  To  a  To  a 

Answered  at  all  Little  soae  Great  Very  Grt 

TO  WHAT  EXTENT  Bxtent  Extent  Ext,nt 

23.  Have  delivery  schedules  taken  priority 

over  quality  decisions?  12(6)  22(10)  64(30)  66(30)  32(15)  12(6) 

24.  Do  engineering  errors  lead  to  problems 

in  managing  your  system/item?  16(7)  9(4)  39(18)  95(44)  37(17)  10(5) 

25.  Do  manufacturing  errors  lead  to  problems 

in  managing  your  system/item?  27(12)  22(10)  46(21)  89(41)  16(7)  5(2) 

26.  Do  quality  assurance  personnel  help  you 
achieve  the  expected  level  of  quality?  18(8) 

25(12) 

27.  Are  you  able  to  influence  the  contractor 

to  provide  quality  products  using 
contract  requirements?  17(8)  1(1) 

28.  Are  you  able  to  influence  the  contractor 

to  provide  quality  products  using  good 
business  practices?  22(10)  4(2)  23(11)  79(36)  61(28)  18(8) 

29.  Would  a  Firm  Fixed  Price  type 
contract  influence  your  contractor 

to  provide  a  quality  product?  25(12)  30(14)  69(32)  54(25)  23(11)  8(4) 

30.  Would  an  award  fee  influence  your 

contractor  to  provide  a  quality  product?  11(5)  25(12)  74(34)  58(27)  10(5) 

29(13) 

31.  Would  a  warranty  influence  your  contractor 

to  provide  a  quality  product?  24(11)  9(4)  29(13)  65(30)  59(27)  20(9) 

32.  Has  your  contractor  used  employee 
motivation  techniques  to  improve 

quality  of  products?  55(25)  9(4)  29(13)  66(30)  30(14)  10(5) 

33.  Has  your  contractor's  top  management's 
attention  influenced  the  quality 

of  the  product  for  your  user?  32(15)  8(4)  23(11)  72(33)  57(26)  7(3) 

34.  Do  your  quality  personnel  focus  on  defect 

prevention  rather  than  detection?  46(21)  11(5)  39(18)  64(30)  37(17)  3(1) 

35.  Do  you  think  the  number  of  quality 
problems  in  the  last  year  were  normal 

for  a  program  of  your  complexity?  32(15)  5(2)  27(12)  62(29)  57(26)  22(10) 

36.  Are  the  right  kind  of  quality  assurance 

personnel  assigned  to  your  program? 37  (  17 )  13(6)  37(17  )  65(30)  41(19)  11  (5) 


30(14)  81(37)  47(22)  6(3) 


22(10)  73(34)  71(33)  23(11) 
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TO  WHAT  EXTENT 


Rot  Not 

Answered  at  all 


very 

Little 


To 

Sone 

Bxtent 


To  a  To  a 
Great  Very  Grt 
Bxtent  Bxtent 


37.  Has  your  prime  contractor  instituted 
an  active  quality  improvement  program 

based  upon  continuous  evaluation?  45(21)  11(5)  29(13)  71(33)  39(18)  7(3) 


38.  Does  your  prime  contractor  conduct 
subcontract  quality  improvement 


programs? 

57(26) 

11(5) 

42(19) 

66(30) 

21(10) 

3(1) 

TO 

WHAT  EXTENT 

Questions  39-42: 

Have  design/manufacturing 
impacted  the  following  on 

Hot 

Answered 

problems 
your  system: 

Hot 

at  all 

Very 

Little 

TO 

Some 

Bxtent 

To  a 
Great 
Bxtent 

To  a 

Very  Grt 
Bxtent 

39 

.  Schedule? 

15(7) 

11(5) 

26(12) 

58(27) 

63(29) 

31(14) 

40 

.  System/item  cost? 

16(7) 

26(12) 

34(16) 

66(30) 

46(21) 

16(7) 

41 

.  Performance? 

17(8) 

25(12) 

55(25) 

54(25) 

43(20) 

9(4) 

42 

.  Reliability? 

19(9) 

27(12) 

43(20) 

58(27) 

40(18) 

14(7) 

TO  WHAT  EXTENT 

43.  Are  "measurable"  quality  improvement 

goals  defined  by  your  organization?34 ( 16 )  28(13)  52(24)  55(25)  22(10)  10(5) 

44.  is  your  contractor’s  performance 
measured  against  quality  standards 

that  you  both  agree  upon?  35(16)  19(9)  24(11)  56(26)  57(26)  11(5) 

45.  Does  your  organization  expend  sufficient 

resources  to  improve  quality?  27(12)  10(5)  50(23)  69(32)  41(19)  7(3) 

46.  Is  producing  a  quality  product  for  your 
user  the  most  important  organizational 

objective?  17(8)  10(5)  21(10)  52(24)  72(33)  35(16) 

47.  Does  your  contractor  share  your  quality 

objectives?  23(11)  2(1)  17(8)  82(38)  62(29)  21(10) 

48.  Do  you  think  you  have  the  right  number  of 
quality  assurance  personnel  assigned  to 

your  program?  27(12)  31(14)  46(21)  45(21)  48(22)  7(3) 

49.  Has  the  staff  quality  organization 
aided  in  helping  raise  the  level  of 

quality  in  your  system/item?  38(18)  37(17)  47(22)  58(27)  18(8)  4(2) 
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TO  WHAT  EXTENT 


Not  Not  Very  To  To  a  To  a 

Answered  at  all  Little  Some  Great  Very  Grt 

Extent  Extent  Extent 


50.  Has  HQ  AFSC  leadership  contributed  to 
raising  the  level  of  quality  in  your 

system/item?  28(13)  58(27)  71(33)  35(16)  12(6)  1(1) 


Questions  51-55: 

To  what  extent  are  you  using  Not  Not  very  To  To  a  to  a 

the  following  measures  and/or  Answered  at  all  Little  Some  Great  Very  Grt 

indicators  of  quality?  Extent  Extent  Extent 

51.  Comparison  of  planned  versus  actual 

man-hours  data.  26(12)  50(23)  32(15)  47(22)  40(18)  9(4) 

52.  A  development  process  that  designs 

quality  into  the  product.  23(11)  27(12)  27(12)  53(24)  59(27)  16(7) 

53.  Monitoring  defects  and  workmanship  data.  26(12)  23(11)  60(28)  49(23)  10(5) 

33(15) 

54.  Field  performance  data.  35(16)  26(12)  16(7)  37(17)  59(27)  27(12) 

55.  Subcontractor  yield  rates.  47(22)  49(23)  37(17)  42(19)  20(9)  5(2) 


Questions  56-60: 

To  what  extent  does  your  contractor 

use  the  following  measures  and/or  Not  Not  very  To  To  a  To  a 

indicators  of  quality?  Answered  at  all  Little  Some  Great  Very  Grt 

Extent  Extent  Extent 

56.  Product  return  rate  (cost  to  rework).  8(4)  25(12)  49(23)  24(11)  9(4) 

79(36) 

57.  Degrees  of  manufacturing  process 

standardization.  73(34)  11(5)  26(12)  47(22)  31(14)  7(3) 

58.  Production  yield  rates.  73(34)  10(5)  23(11)  36(17)  42(19)  9(4) 

59.  Accept/reject  rate  at  work  centers .  71  ( 33  )  8  (  4)  21  (  10  )  39  (  18)  47  (  22)  8(4) 

60.  Material  review  board/quality  deficiency 
reports  (MRB/QDR)  action  effectiveness 

(repeats).  66(30)  9(4)  12(6)  44(20)  47(22)  16(7) 


Questions  61-65: 

To  what  extent  does  your  SPO  use  Hot  Hot  very  To  To  a  To  a 

the  following  approaches  to  Answered  at  all  Little  Some  Great  Very  Grt 

achieve  better  quality?  Extent  Extent  Extent 


61.  Team  building  between  SPO,  CAS, 
contractor . 


and 

22(10)  12(6)  29(13)  67(31)  62(29)  13(6) 


153 


62  . 

63. 
64  . 

65. 


66. 
67  . 
68. 

69. 

70. 


Roe 

Answered 

Producibi lity  risk  reduction  efforts 
funded  well  before  production.  46(21) 

Hardware  quality  audits.  39(18) 

Educational  programs  aimed  at  excellence 
in  management  and  technical  f ields  .  42 ( 19) 

Monitor  performance  to  goals.  39(18) 


Questions  66-70: 

To  what  extent  are  the  following 
approaches  to  achieve  better 
quality  being  used  by  your  Answered 

contractor? 

Total  quality  management  approach.  47(22) 

Quality  improvement  through  design  teams. 

50(23) 

Statistical  process  control  including 
vendors .  65(30) 

Preferred  vendor  program  with  ratings  on 
quality  and  schedule.  61(28) 

Emphasis  on  supplier's  quality.  55(25) 


Not 

Vary 

TO 

To  a 

TO  a 

at  all 

Little 

Some 

Extent 

Great  Very  Grt 
Extent  Extent 

18(8) 

37(17) 

52(24) 

35(16) 

13(6) 

7(3) 

26(12) 

53(24) 

57(26) 

17  (3) 

18(8) 

49(23) 

63(29) 

27(12) 

4(2) 

5(2) 

22(10) 

57(26) 

59(27) 

21(10) 

=  =  =  =  =  = 

======== 

======= 

======= 

====== 

Not 

Vary 

TO 

To  a 

To  a 

at  all 

Little 

Soae 

Extent 

Great 

Extent 

Very  Grt 
Extent 

2(1) 

33(15) 

62(29) 

40(18) 

11(5) 

8(4) 

42(19) 

52(24) 

39(18) 

5(2) 

112(5 

)  34(16) 

56(26) 

19(9) 

8(4) 

17  (3) 

26(12) 

48(22) 

33(15) 

9(4) 

12(6) 

21(10) 

61(28) 

29(13) 

17  (8) 

From  the  following  list  of  10  items  pick  the  three  most 
important  contributors  in  improving  quality  and  productivity 
and  rank  them.  Mark  question  153  as  the  most  important 
contributor,  154  as  the  next  most  important  contributor,  and 
155  as  the  third  most  important  contributor.  For  example,  if 
you  think  program  stability  (item  h'  is  most  important  then  you 
would  mark  question  153h,  the  next  .em  would  go  in  question 
154,  etc. 

Ranking 


a.  Better  job  of  stating  and  controlling  requirements.  4 

b.  Contractor  using  transition  templates  to  do  real  risk  management.  10 

c.  Top-down  corporate  training  to  cultivate  awareness  of  QA  impact  to 

life  cycle  costs.  5 

d.  Commitment  to  not  accepting  poor  quality.  2 

e.  Past  performance  rating  in  source  selection  on  quality.  6 

f.  Require  all  discrepancies  be  referred  to  standards.  9 

g.  Positive  re-enforcement  programs  (incentives).  7 

h.  Program  stability  (funding/requirements/design).  3 

i.  Early  producibility/manuf actur ing/qual ity  input  to  design.  1 

j.  Disallow  all  costs  for  scrap  and  rework  costs  in  pricing  items.  8 


154 


Prom  the  following  list  of  10  items  pick  the  three  most 
important  items  that  detracts  from  achieving  quality  and 
productivity  and  rank  them.  Mark  question  156  as  the  most 
important  detractor  157  as  the  next  most  important  detractor 
and  158  as  the  third  most  important  detractor. 

Ranking 

a.  Unrealistic  program  schedules.  1 

b.  Nebulous  definition  of  quality  assurance.  4 

c.  Lack  of  cooperation  among  personnel  (i.e.,  design,  quality, 

test,  manufacturing,  etc.).  3 

d.  Old  line  quality  approaches  (i.e.,  inspection  quality).  10 

e.  Complexity  of  DoD  organization  which  limits  flexibility  and 

response  time.  6 

f.  Priority  of  quality  within  AF  program  objectives.  5 

g.  Program  instability.  2 

h.  Reassigment  of  key  program  personnel.  8 

i.  Poor  vendor /subcont ractor  quality  levels.  9 

j.  Inadequate  investment  up  front.  7 

THANK  YOU  AGAIN  FOR  YOUR  PARTICIPATION  1 
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APPENDIX  H 

QUALITY  ASSURANCE  MILITARY 
PERSONNEL  DATA 


WORK  PORCE  PROFILE 


Number (Precentage) 


1. 

What  is  your  civilian  rank? 

>  14 

13 

12 

11  < 

09 

2. 

what  is  your  military  rank? 

LTC 

1(4) 

MAJ 

1(4) 

CPT 

5(19) 

1LT 

13(37) 

2LT 

10(37) 

3. 

How  many  years  have  you  worked  in 
quality  assurance? 

>  18 

17-13 

12-8 

7-4 

23(85) 

<  3 
7(7) 

4  . 

How  many  different  systems/items 
have  you  worked  on? 

>  5 
7(26) 

4 

3(11) 

3 

5(19) 

2 

4(15) 

1 

8(30) 

5. 

How  many  years  have  you  worked  on  this 
program? 

>  5 

4 

3 

4(15) 

2 

16(59) 

1 

7(26) 

6. 

Are  you  dedicated  to  a  system/subsystem 
versus  assigned  to  staff? 

YES 

17 (63) 

NO 

13(37) 

7  . 

Your  system/item  is  primarily  in  which 
acquistion  phase? 

Concep 

2(7) 

Val 

3(11) 

PSD 

11(41) 

Prod 

11(41) 

Deploy 

8. 

How  many  full-time  quality  personnel 
are  assigned  to  your  organization? 

>  4 

7(22) 

3 

7(26) 

2 

1(4) 

1 

5(22) 

0 

6(22) 

9. 

How  many  weeks  of  AF  training  in 
quality  assurance  have  you  had? 

>  12 
3(11) 

11-9 

1(4) 

8-6 

5-3 

6(22) 

<  2 
16(59) 

10. 

How  many  courses  that  taught  statistical 
qual ity/pcocess  control  have  you  attended  >4 
within  the  last  5  years?  lT4) 

3 

2 

2(7) 

1 

5(19) 

0 

19(70) 

11. 

How  many  courses  to  improve  product 
performance  and  durability  have  you 
attended  within  the  last  2-3  years  ? 

>  4 
1(4) 

3 

2(7) 

2 

1(4) 

1 

7(26) 

0 

16(59) 

12  . 

What  is  your  highest  level  of  education? 

HS 

HS+ 

Bach 

7(26) 

Bach  + 

15(56) 

Master 

3(11) 

13  . 

what  was  your  college  major? 

ENGR 

20(74) 

SCI 

1(4) 

BUS 

3(11) 

ARTS  OTHER 
1(4) 

14. 

Do  you  have  adequate  opportunities  to 
attend  formal  training  courses? 

YES 

11(41) 

NO 

14(52) 
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Questions  15-24 

Use  the  scale  at  right  to  answer  questions 
15-24.  These  questions  will  help  us  determine 


if  the  correct  subject  material 
our  formal  courses. 

is  taught  in 

Hot 

at  all 

vary 

Little 

TO 

Sone 

Extent 

To  a 
Great 
Extent 

To  a 

Very  Grt 
Extent 

15.  To  what  extent  have  formal 
you  perform  your  job? 

courses  helped 

1(4) 

4(15) 

14(52) 

2(7) 

QUESTIONS  16-23 

To  what  extent  were  the  following 
subjects  covered  in  formal  courses? 

Hot 

at  all 

very 

Little 

TO 

Sone 

Extent 

To  a  To  a 

Great  Very  Grt 
Extent  Extent 

18.  Manufacturing  processes. 

8(30) 

10(37) 

3(11) 

17.  Special  quality  assurance  processes. 

6(22) 

10(37) 

3(11) 

2(7) 

18.  Product  and  hardware  famili 

ar ization . 

10(37) 

6(22) 

3(11) 

1(4) 

1(4) 

19.  State-of-the-art  processing 

technology . 

13(48) 

4(15) 

3(11) 

1(5) 

20.  Computer  systems  including 

software . 

12(44) 

5(19) 

4(15) 

21.  Cost  of  quality  concepts. 

10(37) 

3(11) 

4(15) 

3(11) 

22.  Quality  contract  requirements. 

6(22) 

4(15) 

6(22) 

3(11) 

2(7) 

23.  Testing  procedures. 

9(33) 

3(11) 

3(11) 

4(15) 

2(7) 

24.  How  important  do  you  think  these  topics 
are  to  you  in  performing  your  job? 

1(4) 

1(4) 

4(15) 

6(22) 

9(33) 
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SYSTEM  PROGRAM  QUALITY 

Use  the  scale  at  right  to  answer 

questions  25-28  .  Hot  Answed  Daily  Weakly  Monthly  Qtrly  Waver 

25.  How  often  do  you  interface  with  the 

SPO  program  management  personnel?  18(67)  4(15)  1(4)  3(11)  1(4) 

28.  How  often  do  you  review  user  complaint 

data/trends  on  your  system/item?  1(4)  4(15) 

27.  how  often  do  you  review  your 

contractor's  rework,  repair,  retest 
scrap,  etc.,  levels?  1(4) 

28.  How  often  do  you  review  your 

contractor's  "cost  of  quality"  (failure 
costs  +  routine  inspection  + 
prevention  costs!  data/trends?  1(4)  2(7)  6(22)  18(67) 


3(11)  6(22)  13(48) 


2(7)  11(41)  5(19)  8(30) 
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29.  How  many  quality  of  conformance  problems 
have  had  significant  program  impacts 


during  the  past  year  (i.e.,  schedule,  BK 
cost,  accident,  publicity,  etc.)?  3(11) 

>10 

2(7) 

5 

3(11) 

3 

5(19) 

1 

3(11) 

0 

8(30) 

30.  As  a  percentage  of  total  contract  value, 
what  would  you  estimate  the  "cost  BK  > 

of  quality"  is  for  your  program?  6(22) 

25% 

3(11) 

24-20% 

1(4) 

19-15% 

1(4) 

14-10% 

4(15) 

<  9% 
8(30) 

Use  the  scale  at  right  to  answer  questions 
31-33.  These  questions  are  asking  you  to 
rate  the  breakdown  of  the  cost  of  quality. 

If  you  don't  know,  leave  it  blank.  BK 

>  90% 

89-70% 

69-50% 

49-30% 

_<  29% 

31.  What  percentage  does  failure  cost 

contribute  to  your  cost  of  quality?ll (41) 

1(4) 

1(4) 

2(7) 

3(11) 

32.  What  percentage  does  routine  factory 
inspection  contribute  to  your  cost 
of  quality?  10(37) 

1(4) 

1(4) 

2(7) 

4(15) 

33.  What  percentage  does  prevention  measures 
contribute  to  your  cost  of  quality? 10 ( 37 ) 

1(4) 

1(4) 

2(7) 

4(33) 

34.  What  degree  of  confidence  do  you  have 
in  the  accuracy  for  your  cost  of  BK 

quality  estimate?  13(37) 

>  95% 
2(7) 

94-75% 

3(11) 

74-50% 

2(7) 

49-25% 

1(4) 

<  25% 

333S3=333=3::=3SS3S3S3=::SS333:3S333333S:3S==3333=3::==S3:3=a=:33==:=3S::3==3 

The  next  qroup  of  questions  will  help  us  determine  from  your  perspective 
how  best  to  improve  quality  assurance  management.  Each  question  is 
asking  to  what  extent  you  think  the  statement  applies  to  you/your 


organization.  Therefore,  each  question  should  be  preceded 

by  TO 

WHAT 

EXTENT.  If  you  don't  know  the  answer,  leave 

it  blank. 

Use  the  scale  at  right  to  answer 
questions  35-83. 

Not 

Answered 

Not 

at  all 

very 

Little 

To 

Soee 

Extent 

To  « 
Great 
Extent 

To  a 

Very  Grt 
Extent 

35. 

Does  your  user  influence  contractor 

quality  of  hardware/services? 

1(4) 

2(7) 

4(15) 

2(2) 

1(4) 

36. 

Have  delivery  schedules  taken 

priority 

over  quality  decisions? 

1(4) 

2(7) 

3(11) 

6(22) 

10(37) 

2(7) 

37. 

Do  engineering  errors  lead  to 

problems 

in  managing  your  system/item? 

3(11) 

3(11) 

9(33) 

3(11) 

3(11) 

38. 

Do  manufacturing  errors  lead 

to  problems 

in  managing  your  system/item? 

2(7) 

3(11) 

6(22) 

7(30) 

2(7) 
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TO 

Not 

WHAT  EXTENT  Answered 

Not 

at  all 

Very 

Little 

TO 

Some 

Extent 

To  a 
Great 
Extent 

To  a 
Very  Gr 
Extent 

39. 

Do  quality  assurance  personnel  help 
achieve  the  expected  level  of  quality 

on  your  program?  1(4) 

1(4) 

3(11) 

8(30) 

7(26) 

4(15) 

40. 

Are  you  able  to  influence  the  contractor 
to  provide  quality  products  using 
contract  requirements? 

2(7) 

12(44) 

8(30) 

2(7) 

41  . 

Are  you  able  to  influence  the  contractor 
to  provide  quality  products  using  good 
business  practices? 

2(7) 

4(15) 

11(41) 

5(19) 

2(7) 

42. 

Would  a  Firm  Fixed  Price  type 
contract  influence  your  contractor 
to  provide  a  quality  product?  3(11) 

4(15) 

7(26) 

2(7) 

2(7) 

3(11) 

43  . 

Would  an  award  fee  influence  your 
contractor  to  provide  a  quality  product? 

4(15) 

5(19) 

6(22) 

1(4) 

4(15) 

44  . 

Would  a  warranty  influence  your  contractor 

to  provide  a  quality  product?  1(4) 

2(7) 

5(19) 

6(22) 

5(19) 

4(15) 

45. 

Has  your  contractor  used  employee 
motivation  techniques  to  improve 
quality  of  products?  6(22) 

1(4) 

1(4) 

7(26) 

3(11) 

1(4) 

46  . 

Has  your  contractor's  top  management's 
attention  influenced  the  quality 
of  the  product  for  your  user?  2(7) 

6(22) 

6  27) 

5(19) 

2(7) 

47. 

Do  you  focused  on  defect  prevention 
rather  than  detection?  2(7) 

1(4) 

4(15) 

5(19) 

7(26) 

2(7) 

48. 

Do  you  think  the  number  of  quality 
problems  in  the  last  year  were  normal 
for  a  program  of  your  complexity?  1(4) 

2(7) 

4(15) 

6(22) 

4(15) 

3(11) 

49. 

Are  the  right  kind  of  quality  assurance 
personnel  assigned  to  your  program?  2(7) 

3(11) 

4(15) 

6(22) 

3(11) 

4(15) 

50. 

Has  your  prime  contractor  instituted 
an  active  quality  improvement  program 
based  upon  continuous  evaluation?  2(7) 

2(7) 

3(11) 

5(19) 

6(22) 

2(7) 

51. 

Does  your  prime  contractor  conduct 
subcontract  quality  improvement 
programs?  3(11) 

4(15) 

6(22) 

4(15) 

2(7) 

1(4) 
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{lot 

NOt 

Very 

To 

To  a 

TO  WHAT  EXTENT 

at  all 

Little 

Some 

Extent 

Great 

Extent 

Questions  52-55: 

Have  design/manufacturing 
impacted  the  following  on 

problems 
your  system: 

52.  Schedule? 

1(4) 

1(4) 

7(26) 

6(22) 

53.  System/item  cost? 

3(11) 

1(4) 

2(7) 

6(22) 

6(22) 

54.  Performance? 

2(7) 

3(11) 

1(4) 

6(22) 

6(22) 

55.  Reliability? 

4(15) 

2(7) 

2(7) 

7(26) 

4(15) 

Rot 

Not 

Very 

To 

To  a 

TO  WHAT  EXTENT 

Answered 

at  all 

Little 

Some 

Extent 

Great 

Extent 

56.  Are  "measurable"  quality 

improvement 

==== - ======3= 

=  =  =  =  =  = 

=  =  =  ==  =  = 

goals  defined  by  your  organization?  1(4) 

57.  is  your  contractor's  performance 
measured  against  quality  standards 

7(26) 

10(37) 

3(11) 

1(4) 

that  you  both  agree  upon? 

4(15) 

2(7) 

7(26) 

7(26) 

58.  Does  your  organization  expend  sufficient 

resources  to  improve  quality?  1(4)  4(15)  7(26)  6(22)  4(15) 

59.  Is  producing  a  quality  product  for  your 
user  the  most  important  organizational 


objective? 

1(4) 

2(7) 

8(30) 

8(30) 

2(7) 

60. 

Does  your  contractor  share  your  quality 

objectives? 

4(15) 

1(4) 

2(7) 

9(33) 

4(15) 

61 . 

Do  you  think  you 

have  the  right  number  of 

quality  assurance 

personnel  assigned  to 

your  program? 

1(4) 

7(26) 

6(22) 

4(15) 

4(15) 

62.  Has  the  staff  quality  organization 
aided  in  helping  raise  the  level  of 

quality  in  your  system/item?  1(4)  3(11)  9(33)  5(19)  3(11) 

63.  Has  HQ  AFSC  leadership  contributed  to 
raising  the  level  of  quality  in  your 

system/item?  1(4)  9(33)  10(37)  2(7) 
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To  a 

Very  Grt 
Extent 


6(22) 

5(19) 

4(15) 

4(15) 

To  a 

Very  Grt 
Extent 

=3333== 

1(4) 

2(7) 

1(4) 

3(11) 

2(7) 

1(4) 

1(4) 


=  =  = 

II 

1! 

II 

II 

II 

II 

II 

II 

It 

II 

II 

If 

II 

II 

II 

II 

II 

II 

II 

II 

II 

II 

II 

II 

II 

Questions  64-68: 

To  what  extent  are  you  using 
following  measures  and/or 

the 

Hot 

Hot 

Very 

To 

To  a 

To  a 

indicators  of  quality? 

Answered 

at  all 

Little 

Some 

Great 

Very  Grt 

latent 

Extent 

Extent 

64  . 

Comparison  of  planned  versus 
man-hours  data. 

actual 

1(4) 

10(37) 

2(7) 

6(22) 

1(4) 

2(7) 

65  . 

A  development  process  that  de 
quality  into  the  product. 

signs 

1(4) 

5(19) 

1(4) 

10(37) 

4(15) 

1(4) 

66. 

Monitoring  defects  and  workmanship 

data . 

3(11) 

6(22) 

7(26) 

5(19) 

1(4) 

67  . 

Field  performance  data. 

1(4) 

4(15) 

2(7) 

6(22) 

9(33) 

1(4) 

68  . 

Subcontractor  yield  rates. 

2(7) 

8(30) 

3(11) 

6(22) 

1(4) 

1(4) 

®  *  = 

Questions  69-73: 

To  what  extent  does  your  contractor-  „ 
use  the  following  measures  Answered 

Hot 

at  all 

Very 

Little 

To 

Some 

To  a 
Great 

To  a 

Very  Grt 

and/or  indicators  of  quality? 

Extent 

Extent 

Extent 

69. 

Product  return  rate  (cost  to 

rework )  . 

2(7) 

2(7) 

3(11) 

5(19) 

1  US) 

70. 

Degrees  of  manufacturing  proc 
standardization , 

ess 

7(26) 

3(11) 

1(4) 

5(19) 

4(15) 

71  . 

Production  yie  :  races. 

5(22) 

5(19) 

1(4) 

2(7) 

4(15) 

2(7) 

72  . 

Accept /re  j  ect  .te  at  woi  <  ce 

nters  . 

6(22) 

3(11) 

1(4) 

3(11) 

4(15) 

2(7) 

73  . 

MRS/QDR  action  effectiveness 

( repeats )  . 

2(7) 

1(4) 

3(11) 

5(19) 

3(11) 

6(22) 


Questions  74-78: 


To  what  extent  does  your  SPO 
use  the  following  approaches 
to  achieve  better  quality? 

Not 

Answered 

Not 

at  all 

Very 

Little 

TO 

Some 

Extent 

To  a 
Great 
Extent 

To  a 

very  Grt 
Extent 

74  . 

Team  building  between  SPO,  CAS,  and 
contractor .  1(4) 

4(15) 

3(11) 

10(37) 

4(15) 

2(7) 

75. 

Pr oducib i 1 i ty  risk  reduction 
funded  well  before  production 

efforts 

4(15) 

7(26) 

6(22) 

6(22) 

1(4) 

76  . 

Hardware  quality  audits. 

2(7) 

5(19) 

3(11) 

4(15) 

6(22) 

3(11) 

77  . 

Educational  programs  aimed  at 
in  management  and  technical  f 

excellence 
ields .3(11) 

6(22) 

6(22) 

5(19) 

1(4) 

1(4) 

78  . 

Monitor  performance  to  goals. 

3(11) 

4(15) 

3(11) 

6(22) 

3(11) 

3(11) 
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From  the  following  list  of  10  items  pick  the  three  most 
important  contributors  in  improving  quality  and  productivity 
and  rank  them.  Mark  question  153  as  the  most  important 
contributor,  154  as  the  next  most  important  contributor,  and 
155  as  the  third  most  important  contributor.  For  example,  if 
you  think  program  stability  (item  h)  is  most  important  then  you 
would  mark  question  153h,  the  next  item  would  go  in  question 
154,  etc. 

Ranking 

Mil/Civ 


a.  3etter  job  of  stating  and  controlling  requirements.  3 

b.  Contractor  using  transition  templates  to  do  real  risk  management.  9 

c.  Top-down  corporate  training  to  cultivate  awareness  of  QA  impact  to 

life  cycle  costs.  4 

d.  Commitment  to  not  accepting  poor  quality.  1 

e.  Past  performance  rating  in  source  selection  on  quality.  7 

f.  Require  all  discrepancies  be  referred  to  standards.  10 

g.  Positive  re-enforcement  programs  (incentives).  8 

h.  Program  stability  ( f und ing/ r equ i r ement s/des ign ) .  5 

i.  Early  producibility/manuf actur ing/qual ity  input  to  aesign.  2 

j.  Disallow  all  costs  for  scrap  and  rework  costs  in  pricing  items.  6 
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Prom  the  following  list  of  10  items  pick  the  three  most 

important  items  that  detracts  from  achieving  quality  and 

productivity  and  rank  them.  Mark  question  156  as  the  most 

important  detractor  157  as  the  next  most  important  detractor 

and  158  as  the  third  most  important  detractor.  Ranking 

Mil/Cil 

a.  Unrealistic  program  schedules.  1 

b.  Nebulous  definition  of  quality  assurance.  6 

c.  Lack  of  cooperation  among  personnel  (i.e.,  design,  quality, 

test,  manufacturing,  etc.).  3 

d.  Old  line  quality  approaches  (i.e.,  inspection  quality).  5 

e.  complexity  of  DoD  organization  whic  limits  flexibility  and 

response  time.  7 

f.  Priority  of  quality  within  AF  program  objectives.  2 

g.  Program  instability.  9 

h.  Reassigment  of  key  program  personnel.  10 

i.  Poor  vendor/subcontractor  quality  levels.  4 

j.  inadequate  investment  up  front.  8 


THANK  YOU  AGAIN  FOR  YOUR  PARTICIPATION  I 
SAMPLE  SIZE  27 
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APPENDIX  I 

QUALITY  ASSURANCE  CIVILIAN 
PERSONNEL  DATA 


WORK  FORCE  PROFILE 


Number (Percentage) 


1 . 

What  is  your  civilian  rank? 

>  14 
476) 

13 

25(37) 

12 

30(45) 

11 

3(5) 

<  09 
578) 

2  . 

What  is  your  military  rank? 

LTC 

MAJ 

CPT 

1LT 

2LT 

3. 

How  many  years  have  you  worked  in 
quality  assurance? 

>  18 

17(25) 

17-13 

8(12) 

12-8 

13(19) 

7-4 

16(24) 

<  3 

13(19) 

4  . 

How  many  different  systems/items 
have  you  worked  on? 

>  5 

?3 (79) 

4 

2(3) 

3 

6(9) 

2 

3(5) 

1 

2(3) 

5. 

How  many  years  have  you  worked  on  this 
program? 

>  5 

II (16  ) 

4 

18(27) 

3 

7(10) 

2 

19(28) 

1 

9(13) 

6. 

Are  you  dedicated  to  a  system/subsystem 
versus  assigned  to  staff? 

YES 

45(67) 

NO 

21(31) 

7  . 

Your  system/item  is  primarily  in  which 
acquistion  phase? 

Concep 

1(2) 

Val 

8(12) 

FSD 

29(43) 

Prod 

19(28) 

Deploy 

4(6) 

8. 

How  many  full-time  quality  personnel 
are  assigned  to  your  organization? 

>  4 

29  (  43  ) 

3 

10(15) 

2 

12(18) 

1 

10(15) 

0 

3(5) 

9  . 

How  many  weeks  of  AF  training  in 
quality  assurance  have  you  had? 

>  12 
21(31) 

11-9 

5(8) 

8-6 

10 (15) 

5-3 

12(18) 

<  2 

18(27) 

10. 

How  many  courses  that  taught  statistical 
quality/process  control  have  you  attended  >4 
within  the  last  5  years?  6T 9 ) 

3 

3(5) 

2 

6(9) 

1 

27(40) 

0 

25(37) 

11 . 

How  many  courses  to  improve  product 
performance  and  durability  have  you 
attended  within  the  last  2-3  years  ? 

>  4 

Tl (16) 

3 

6(9) 

2 

11(16) 

1 

10(15) 

0 

29  (43  ! 

12  . 

what  is  your  highest  level  of  education? 

HS 

HS  + 

16 (24) 

Bach 

7(10) 

Bach+  Master 
29(43)  13(19) 

13  . 

what  was  your  college  major? 

ENGR 

45(67) 

SCI 

9(13) 

BUS 

7(10) 

ARTS 

OTHER 

2(3) 

14  . 

Do  you  have  adequate  opportunities  to 
attend  formal  training  courses? 

YES 

43(64) 

NO 

21(31) 
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Questions  15-24 

Use  the  scale  at  right  to  answer  questions 

15-24.  These  questions  will  help  us  determine 
if  the  correct  subject  material  is  taught  in  Hot 

our  formal  courses.  at  all 

15.  To  what  extent  have  formal  courses  helped 

you  perform  your  job?  4(6) 

3==asa: 

very 

Little 

11(16) 

ZS3S3=3S 

To 

Soae 

Extent 

25(37) 

To  a  To  a 

Great  Very  Grt 
Extent  Extent 

19(28)  4(6) 

QUESTIONS  16-23 

To  what  extent  were  the  following 
subjects  covered  in  formal  courses? 

Hot 

at  all 

Very 

Little 

To 

Soae 

Extent 

To  a  To  a 

Great  Very  Grt 
Extent  Extent 

16 . 

Manufacturing  processes. 

14(21) 

21(31) 

16(24) 

11(16) 

17  . 

Special  quality  assurance  processes. 

8(12) 

25(37) 

18(27) 

8(12) 

1(2) 

18. 

Product  and  hardware  familiarization. 

20(30) 

17(25) 

21(31) 

3(5) 

19. 

State-of-the-art  processing  technology. 

21(31) 

24(35) 

14(21) 

2(3) 

1(2) 

20  . 

Computer  systems  including  software. 

16(23) 

27(40) 

13(19) 

5(8) 

1(2) 

21  . 

Cost  of  quality  concepts. 

11(16) 

25(37) 

19(28) 

6(9) 

1(2) 

22. 

Quality  contract  requirements. 

5(8) 

18(27) 

20(30) 

16(24) 

4(6) 

23  . 

Testing  procedures. 

9(13) 

25(37) 

16(24) 

11(16) 

24. 

How  important  do  you  think  these  topics 
are  to  you  in  performing  your  job? 

1(2) 

1(2) 

15(22) 

30(45) 

17(25) 

SYSTEM  PROGRAM  QUALITY 

Use  the  scale  at  right  to  answer 

questions  25-28.  jlot  Ansved  Dally 

Weekly 

Monthly  Qtrly 

Waver 

25  . 

How  often  do  you  interface  with  the 

SPO  program  management  personnel? 

38 (57) 

15(22) 

9(13) 

2(3) 

1(2) 

26. 

How  often  do  you  review  user  complaint 
data/trends  on  your  system/item?  3(5) 

5(8) 

7(10) 

23(34) 

13(19) 

11(16) 

27. 

How  often  do  you  review  your 
contractor's  rework,  repair,  retest 
scrap,  etc.,  levels?  2(3) 

2(3) 

1(2) 

28(42) 

17(25) 

12(18) 

28  . 

How  often  do  you  review  your 
contractor's  "cost  of  quality"  (failure 
costs  +  routine  inspection  + 
prevention  costs")  data/trends?  2(3) 

1(2) 

1(2) 

17(25) 

17(25) 

24(36) 
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How  many  quality  of  conformance  problems 
have  had  significant  program  impacts 
during  the  past  year  (i.e.,  schedule, 

>10 

5 

3 

1 

0 

cost,  accident,  publicity,  etc.)? 

2(3) 

7(10) 

9(13) 

16(24) 

10(15) 

16(24) 

As  a  percentage  of  total  contract 
what  would  you  estimate  the  "cost 

value , 

BK 

>  25% 

24-20% 

19-15% 

14-10% 

<  9% 

of  quality"  is  for  your  program? 

6(9) 

4(6) 

6(9) 

7(10) 

11(16) 

22(33) 

s  s  s  3  s  s  s 

3  3  3  3  3  3  3 

Use  the  scale  at  right  to  answer  questions 
31-33.  These  questions  are  asking  you  to 
rate  the  breakdown  of  the  cost  of  quality. 

If  you  don't  know,  leave  it  blank.  BK  _>  90%  89-70%  69-50%  49-30%  £  29% 

31.  What  percentage  does  failure  cost 


contribute  to  your  cost  of  quality?23 ( 34 ) 

3(5) 

3(5) 

5(8) 

4(6) 

14(21) 

32  . 

What  percentage  does  routine  factory 
inspection  contribute  to  your  cost 
of  quality?  23(34) 

1(2) 

1(2) 

6(9) 

9(13) 

11(16) 

33  . 

What  percentage  does  prevention  measures 
contribute  to  your  cost  of  quality?23 ( 34 ) 

1(2) 

5(8) 

5(8) 

17(25) 

34  . 

What  degree  of  confidence  do  you  have 
in  the  accuracy  for  your  cost  of  BK 

quality  estimate?  21(31) 

>  95% 
2(3) 

94-75% 

4(6) 

74-50% 

9(13) 

49-25% 

1(2) 

<  25% 
15(22) 

====================== 


The  next  qroup  of  questions  will  help  us  determine  from  your  perspective 
how  best  to  improve  quality  assurance  management.  Each  question  is 
asking  to  what  extent  you  think  the  statement  applies  to  you/your 
organization.  Therefore,  each  question  should  be  preceded  by  TO  WHAT 
EXTENT.  If  you  don't  know  the  answer,  leave  it  blank. 

Use  the  scale  at  right  to  answer 
questions  35-83. 


Hot 

Answered 


35.  Does  your  user  influence  contractor 

quality  of  hardware/services?  1(2) 

36.  Have  delivery  schedules  taken  priority 

over  quality  decisions?  4(6) 

37.  do  engineering  errors  lead  to  problems 

in  managing  your  system/item?  6(9) 

38.  Do  manufacturing  errors  lead  to  problems 

in  managing  your  system/item?  7(10) 

TO  WHAT  EXTENT 


Hot 

at  all 

very 

Little 

TO 

Sone 

Extent 

To  a 
Great 
Extent 

To  a 

Very  Grt 
Extent 

6(9) 

13(19) 

29(43) 

10(15) 

2(3) 

5(8) 

3(5) 

17(25) 

15(22) 

17(25) 

17(25) 

19(28) 

14(21) 

7(10) 

2(3) 

5(8) 

34(51) 

7(10) 

8(12) 
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Not  Not  Very  To  To  a  To  a 

Answered  at  all  Little  Some  Great  Very  Grt 

39.  Do  quality  assurance  personnel  help  Extent  Extent  Extent 

achieve  the  expected  level  of  quality 

on  your  program?  2(3)  5(3)  27(40)  23(34)  6(9) 

40.  Are  you  able  to  influence  the  contractor 
to  provide  quality  products  using 

contract  requirements?  2(3)  2(3)  31(46)  20(30)  7(10) 

41.  Are  you  able  to  influence  the  contractor 
to  provide  quality  products  using  good 

business  practices?  4(6)  2(3)  11(16)  28(42)  10(15)  5(8) 

42.  would  a  Firm  Fixed  Price  type 
contract  influence  your  contractor 

to  provide  a  quality  product?  3(5)  4(6)  23(34)  21(31)  6(9)  6(9) 

43.  Would  an  award  fee  influence  your 

contractor  to  provide  a  quality  product?  3(5)  7(10)  26(39)  19(28)  2(3) 

4(6) 

44.  Would  a  warranty  influence  your  contractor 

to  provide  a  quality  product?  4(6)  3(5)  11(16)  19(28)  19(28)  8(12) 

45.  Has  your  contractor  used  employee 
motivation  techniques  to  improve 

quality  of  products?  3(5)  6(9)  8(11)  27(40)  10(15)  3(5) 

46.  Has  your  contractor's  top  management's 
attention  influenced  the  quality 

of  the  product  for  your  user?  4(6)  1(2)  12(18)  27(40)  12(18)  3(5) 

47.  Do  you  focused  on  defect  prevention 

rather  than  detection?  4(6)  3(5)  3(5)  13(19)  33(49)  7(10) 

48.  Do  you  think  the  number  of  quality 
problems  in  the  last  year  were  normal 

for  a  program  of  your  complexity?  2(3)  6(9)  6(9)  24(36)  17(25)  4(6) 

49.  Are  the  right  kind  of  quality  assurance 

personnel  assigned  to  your  program?  4(6)  15(22)  22(33)  14(21)  7(10) 

50.  as  your  prime  contractor  instituted 
an  active  quality  improvement  program 

based  upon  continuous  evaluation?  6(9)  4(6)  11(16)  18(27)  15(22)  4(6) 

51.  Does  your  prime  contractor  conduct 
subcontract  quality  improvement 

programs?  8(12)  3(5)  10(15)  21(31)  15(22) 
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Not 

Not 

Very 

TO 

To  a 

To  a 

TO  WHAT  EXTENT 

Answered 

at  all 

Little 

Some 

Bxtent 

Great 

Extent 

Very  Grt 

Extent 

Questions  52-55: 

Have  design/manufacturing  problems 


impacted  the  following  on  your 

system: 

52  . 

Schedule? 

2(3) 

4(6) 

3(5) 

18(27) 

22(33) 

13(19) 

53  . 

System/item  cost? 

2(3) 

6(9) 

6(9) 

21(31) 

17(25) 

9(13) 

54  . 

Per  f ormance? 

1(2) 

8(12) 

7(10) 

26(39) 

15(22) 

5(8) 

55. 

Reliability? 

1(2) 

7(10) 

9(13) 

20(30) 

18(27) 

7(10) 

Not 

Not 

Very 

To 

To  a 

To  a 

Answered 

•t  all 

Little 

Some 

Great 

Very  Grt 

TO 

WHAT  EXTENT 

Bxtent 

Extent 

Bxtent 

56  . 

Are  "measurable"  quality  improvement 

goals  defined  by  your  organization? 

12(18) 

14(21) 

17(25) 

13(19) 

4(6) 

57. 

Is  your  contractor's  performance 

measured  against  quality  standards 

that  you  both  agree  upon? 

1(2) 

10(15) 

10(15) 

22(33) 

14(21) 

3(21) 

58  . 

Does  ycur  organization  expend 

sufficient 

resources  to  improve  quality? 

1(2) 

7(10) 

12(18) 

26(39) 

14(21) 

2(3) 

59. 

Is  producing  a  quality  product 

for  your 

user  t ne  most  important  organi 

zational 

object i ve? 

1(2) 

5(3) 

9(13) 

19(28) 

13(19) 

13(19) 

60.  Does  y’ur  contractor  share  your  quality 

objectives?  1(2)  2(3)  11(16)  30(45)  16(24)  1(2) 

61.  Do  you  think  you  have  the  right  number  of 
qualit"  assurance  personnel  assigned  to 

your  f  ogram?  1(2)  11(16)  15(22)  15(22)  16(24)  3(5) 

62.  Has  th“  staff  quality  organization 
aided  in  helping  raise  the  level  of 

quality  in  your  system/item?  2(3)  8(12)  11(16)  26(39)  14(21)  1(2) 

63.  Has  HQ  AFSC  leadership  contributed  to 
raisin  i  the  level  of  quality  in  your 

system  'item?  1  (  2  )  12  (  18  )  24  (  36  )  18  (  27  )  7  (  10  )  1  (  2) 
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Questions  64-68: 


To  what  extent  ace  you  using 

the  following  measures  and/or  Hot 

indicators  of  quality?  Answered 

Hot 

at  all 

Very 

Little 

To 

Soae 

Bxtent 

To  a 
Great 
Bxtent 

To  a 

Very  Grt 
Bxtent 

64.  comparison  of  planned  versus  actual 
man-hours  data. 

4(6) 

18(27) 

12(18) 

19(28) 

5(8) 

2(3) 

65.  A  development  process  that  designs 
quality  into  the  product. 

5(8) 

10(15) 

24(36) 

20(30) 

2(3) 

66.  Monitoring  defects  and  workmanship 

data . 

3(5) 

7(10) 

17(25) 

27(40) 

8(12) 

67.  Field  performance  data. 

5(8) 

4(6) 

3(5) 

19(28) 

21(31) 

11(16) 

68.  Subcontractor  yield  rates. 

3(5) 

14 (21) 

8(12) 

22(33) 

19(15) 

2(3) 

333333333333333333333333333333333333333 

3333333 

*  -  *  -  = 

Questions  69-73: 

=2  3  »  =  =  —  = 

_  — 

====== 

To  what  extent  does  your  contractor 

use  the  following  measures  and/or  Rot 

Hot 

Very 

To 

To  a 

To  a 

indicators  of  quality? 

Answered 

at  all 

Little 

Soae 

Great 

Very  Grt 

Bxtent 

Bxtent 

Bxtent 

69.  Product  return  rate  (cost  to 

rework )  . 

1(2) 

9(13) 

26(39) 

12(18) 

8(12) 

70.  Degrees  of  manufacturing  process 

standardization . 

8(12) 

7(10) 

6(9) 

24(36) 

11(16) 

1(2) 

71.  Production  yield  rates. 

8(12) 

4(6) 

5(8) 

19(28) 

17(25) 

3(5) 

72.  Accept/re ject  rate  at  work  centers  .  7  (10 ) 

1(2) 

3(5) 

27(40) 

14(21) 

4(6) 

73.  MRB/QDR  action  effectiveness 

( repeats)  . 

2(3) 

5(8) 

25(37) 

13(19) 

4(6) 

8(12) 

333S333333333SS3S3333333333S33333 

3333333 

3333333 

333333 

33333  3 

Questions  74-78: 

To  what  extent  does  your  SPO 

Hot 

Hot 

very 

To 

To  a 

To  a 

use  the  following  approaches 

Answered 

at  all 

Little 

Soae 

Great 

Very  Grt 

to  achieve  better  quality? 

Bxtent 

Bxtent 

Bxtent 

74.  Team  building  between  SPO,  CAS,  and 

contractor . 

3(5) 

2(3) 

18 (27) 

16(24) 

18(27) 

5(8) 

75.  Producibility  risk  reduction 

efforts 

funded  well  before  production 

3(5) 

6(9) 

9(13) 

24(36) 

17(25) 

4(6) 

76.  Hardware  quality  audits. 

5(8) 

1(2) 

4(6) 

24(36) 

21(31) 

7(10) 

77.  Educational  programs  aimed  at 

excellence 

in  management  and  technical  fields. 5(3) 

9(13) 

12(18) 

21(31) 

13(19) 

1(2) 

78.  Monitor  performance  to  goals. 

2(3) 

2(3) 

8(12) 

20(30) 

24(36) 

6(9) 
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Questions  79-83: 

To  what  extent  are  the  following 


approaches  to  achieve  better 
quality  being  used 
by  your  contractor? 

Rot 

Answered 

ROt 

at  all 

vary 

Little 

TO 

Soae 

Extant 

to  a  To  a 

Great  Very  Grt 
Extent  Extent 

79. 

Total  quality  management  approach.  5(8) 

10(15) 

30(45) 

10(15) 

4(6) 

80  . 

Quality  improvement  through 

design  teams. 

6(9) 

13(19) 

19(28) 

17(25) 

1(2) 

3(5) 

81 . 

Statistical  process  control 

including 

vendors . 

5(8) 

6(9) 

17  (25) 

22(32) 

8(12) 

1(2) 

82  . 

Preferred  vendor  program  with  ratings  on 

quality  and  schedule. 

5(8) 

2(3) 

10(15) 

21(31) 

18(27) 

3(5) 

83. 

Emphasis  on  supplier's  quali 

ty.  6(9) 

1(2) 

7(10) 

24(36) 

16(24) 

3(5) 

From  the  following  list  of  10  items  pick  the  three  most 
important  contributors  in  improving  quality  and  productivity 
and  rank  them.  Mark  question  153  as  the  most  important 
contributor,  154  as  the  next  most  important  contributor,  and 
155  as  the  third  most  important  contributor.  For  example,  if 
you  think  program  stability  (item  h)  is  most  important  then  you 
would  mark  question  153h,  the  next  item  would  go  in  question 
154,  etc. 

Ranking 

Mil/Cil 


a.  Better  job  of  stating  and  controlling  requirements.  3 

b.  Contractor  using  transition  templates  to  do  real  risk  management.  9 

c.  Top-down  corporate  training  to  cultivate  awareness  of  QA  impact  to 

life  cycle  costs.  4 

d.  commitment  to  not  accepting  poor  quality.  1 

e.  past  performance  rating  in  source  selection  on  quality.  7 

f.  Require  all  discrepancies  be  referred  to  standards.  10 

g.  Positive  re-enforcement  programs  (incentives).  8 

h.  Program  stability  (funding/requirements/design).  5 

i.  Early  producibility/manuf actur ing/qual ity  input  to  design.  2 

j.  Disallow  all  costs  for  scrap  and  rework  costs  in  pricing  items.  6 


From  the  following  list  of  10  items  pick  the  three  most 

important  items  that  detracts  from  achieving  quality  and 

productivity  and  rank  them.  Mark  question  156  as  the  most 

important  detractor  157  as  the  next  most  important  detractor 

and  158  as  the  third  most  important  detractor.  Ranking 

Mil/Civ 

a.  Unrealistic  program  schedules.  1 

b.  Nebulous  definition  of  quality  assurance.  6 

c.  Lack  of  cooperation  among  personnel  (i.e.,  design,  quality, 

test,  manufacturing,  etc.).  3 

d.  Old  line  quality  approaches  (i.e.,  inspection  quality).  5 

e.  Complexity  of  DoD  organization  which  limits  flexibility  and 

response  time.  7 

f.  Priority  of  quality  within  AF  program  objectives.  2 

g.  Program  instability.  9 

h.  Reassigment  of  key  program  personnel.  10 

i.  Poor  vendor/subcontractor  quality  levels.  4 

j.  Inadequate  investment  up  front.  8 

THANK  YOU  AGAIN  FOR  YOUR  PARTICIPATION  1 
SAMPLE  SIZE  67 
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